What is SettingWithCopyWarning in Pandas?
Why does SettingWithCopyWarning occur?
To demonstrate why
SettingWithCopyWarning occurs, consider the following DataFrame:
dfA0 31 4
Let us try to modify values via chaining:
We encounter a
SettingWithCopyWarning because the
head(~) method performs slicing behind the scenes, which means that it is unclear whether a copy or a view of the DataFrame is returned.
To briefly explain the difference between a copy and a view:
a copy of a DataFrame is a new DataFrame and modifying it would have no effect on the source DataFrame and vice versa.
a view of a DataFrame shares the same memory address as the source DataFrame, and so modifying the view will also modify the source DataFrame and vice versa.
SettingWithCopyWarning tells us that the original DataFrame may or may not be affected. In this case, by inspecting our DataFrame
df, we find that the value has been updated as well:
dfA0 91 4
Now, instead of setting the value to an integer like
9, let's update the value to a floating number like
Notice how the original
df has been kept intact without any updates. The only difference between this case and the previous case is that we tried updating our column value with a floating value (
9.99) instead of an integer (
The reason this happens is that updating the column type will cause Pandas to create a new copy of the DataFrame. In this case, column
A is of type
int64; updating a value in this column with another integer would not change the type of column
A, and hence Pandas would directly modify the original DataFrame. In other words, the view of the DataFrame is updated. However, updating a value with a float would change the type of the column from integer to float. When this happens, Pandas creates a copy of the DataFrame and updates this copy instead of the original DataFrame.
Whether a view or a copy is updated can obviously lead to vastly different and often unexpected results, so Pandas plays it safe by always throwing a
SettingWithCopyWarning if you try to update a slice of a DataFrame.
There is a widespread misconception about views. Whether or not the original DataFrame will be mutated does not depend only on whether the method returns a copy or a view. In fact, if you check the
_is_view property of the
head(~) method, you will get a
This output is somewhat misleading because this suggests that changes made to the returned DataFrame of
head(2), which is a view of
df, will also propagate to the original DataFrame
df. However, mutations are propagated only if the column type does not change as we have demonstrated.
Whenever possible, avoid chaining methods when performing an assignment. For instance, for the above case, just directly use
iloc to update the values:
df.iloc[0,0] = 10dfA B0 10 61 4 72 5 8
Using loc and iloc for slicing
To demonstrate that changes made to slices are propagated to the original DataFrame:
iloc[:,:] means we want all the rows and column, that is, we want the entire DataFrame. Next, we are using
iloc again to update the value of the cell at
9.9. The key here is that, unlike
iloc will always return a view regardless of whether the column type changes, and so the original DataFrame is always mutated.
Creating a copy
To avoid the confusion of whether changes are propagated to the original DataFrame altogether, we can always create a copy using the
This ensures that you are always dealing with a copy instead of a view of a DataFrame.