search
Search
Join our weekly DS/ML newsletter layers DS/ML Guides
menu
menu search toc more_vert
Robocat
Guest 0reps
Thanks for the thanks!
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
help Ask a question
Share on Twitter
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to
A
A
brightness_medium
share
arrow_backShare
Twitter
Facebook

What is SettingWithCopyWarning in Pandas?

Pandas
chevron_right
Common questions
schedule Jul 1, 2022
Last updated
local_offer PythonPandas
Tags

Why does SettingWithCopyWarning occur?

To demonstrate why SettingWithCopyWarning occurs, consider the following DataFrame:

df = pd.DataFrame({"A":[3,4]})
df
A
0 3
1 4

Let us try to modify values via chaining:

df.head(2).iloc[0,0] = 9
1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

We encounter a SettingWithCopyWarning because the head(~) method performs slicing behind the scenes, which means that it is unclear whether a copy or a view of the DataFrame is returned.

NOTE

To briefly explain the difference between a copy and a view:

  • a copy of a DataFrame is a new DataFrame and modifying it would have no effect on the source DataFrame and vice versa.

  • a view of a DataFrame shares the same memory address as the source DataFrame, and so modifying the view will also modify the source DataFrame and vice versa.

The SettingWithCopyWarning tells us that the original DataFrame may or may not be affected. In this case, by inspecting our DataFrame df, we find that the value has been updated as well:

df
A
0 9
1 4

Now, instead of setting the value to an integer like 9, let's update the value to a floating number like 9.9:

df = pd.DataFrame({"A":[3,4]})
df.head(2).iloc[0,0] = 9.9
A
0 3
1 4

Notice how the original df has been kept intact without any updates. The only difference between this case and the previous case is that we tried updating our column value with a floating value (9.99) instead of an integer (9).

The reason this happens is that updating the column type will cause Pandas to create a new copy of the DataFrame. In this case, column A is of type int64; updating a value in this column with another integer would not change the type of column A, and hence Pandas would directly modify the original DataFrame. In other words, the view of the DataFrame is updated. However, updating a value with a float would change the type of the column from integer to float. When this happens, Pandas creates a copy of the DataFrame and updates this copy instead of the original DataFrame.

Whether a view or a copy is updated can obviously lead to vastly different and often unexpected results, so Pandas plays it safe by always throwing a SettingWithCopyWarning if you try to update a slice of a DataFrame.

WARNING

There is a widespread misconception about views. Whether or not the original DataFrame will be mutated does not depend only on whether the method returns a copy or a view. In fact, if you check the _is_view property of the head(~) method, you will get a True:

df = pd.DataFrame({"A":[3,4]})
df.head(2)._is_view
True

This output is somewhat misleading because this suggests that changes made to the returned DataFrame of head(2), which is a view of df, will also propagate to the original DataFrame df. However, mutations are propagated only if the column type does not change as we have demonstrated.

Possible fixes

Avoid chaining

Whenever possible, avoid chaining methods when performing an assignment. For instance, for the above case, just directly use iloc to update the values:

df.iloc[0,0] = 10
df
A B
0 10 6
1 4 7
2 5 8

Using loc and iloc for slicing

Instead of slicing the DataFrame with methods like head(~), you can use loc and iloc which always return a view.

To demonstrate that changes made to slices are propagated to the original DataFrame:

df.iloc[:,:].iloc[0,0] = 9.9
df
A
0 9.90
1 4.00

Here, iloc[:,:] means we want all the rows and column, that is, we want the entire DataFrame. Next, we are using iloc again to update the value of the cell at 0,0 to 9.9. The key here is that, unlike head(~), iloc will always return a view regardless of whether the column type changes, and so the original DataFrame is always mutated.

Creating a copy

To avoid the confusion of whether changes are propagated to the original DataFrame altogether, we can always create a copy using the copy(deep=True) method:

df = pd.DataFrame({"A":[3,4]})
df_copy = df.head(2).copy(deep=True) # Create a copy
df_copy.iloc[0,0] = 9 # This does not affect the original df

This ensures that you are always dealing with a copy instead of a view of a DataFrame.

mail
Join our newsletter for updates on new DS/ML comprehensive guides (spam-free)
robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
Ask a question or leave a feedback...
0
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!