Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependencies, drop py35, test in py310, improve itemadapter performance #66

Merged
merged 4 commits into from
Mar 22, 2022

Conversation

elacuesta
Copy link
Member

@elacuesta elacuesta commented Mar 19, 2022

  • Drop Python 3.5 support
  • Add Python 3.10 support
  • Bump minimum dependencies: Scrapy 1.6 (2019-01-30), python-scrapinghub 2.1.0 (2019-01-14)
  • Improve performance by updating itemadapter usage
  • Remove six as test dependency

Calling itemadapter.ItemAdapter.is_item directly instead of the itemadapter.is_item alias (which imports itemadapter.ItemAdapter on each call) is faster:

from timeit import timeit
from itemadapter import is_item, ItemAdapter
from scrapy.item import Item

def test_dict_alias():
    return is_item({})

def test_dict_direct():
    return ItemAdapter.is_item({})

def test_item_alias():
    return is_item(Item())

def test_item_direct():
    return ItemAdapter.is_item(Item())

print("test_dict_alias: ", timeit(test_dict_alias, number=10**6))
print("test_dict_direct:", timeit(test_dict_direct, number=10**6))
print("test_item_alias: ", timeit(test_item_alias, number=10**6))
print("test_item_direct:", timeit(test_item_direct, number=10**6))
test_dict_alias:  2.2290923200000003
test_dict_direct: 1.314303395
test_item_alias:  3.5215694760000003
test_item_direct: 2.772791269

(At scrapy/itemadapter@1203b5e, with a 2020 Macbook Pro, Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz)

See also scrapy/itemadapter#59 and the issues/PRs linked there.

@codecov
Copy link

codecov bot commented Mar 19, 2022

Codecov Report

Merging #66 (20967f1) into master (9c0c6f3) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head 20967f1 differs from pull request most recent head 2c2ec56. Consider uploading reports for the commit 2c2ec56 to get more accurate results

@@            Coverage Diff             @@
##           master      #66      +/-   ##
==========================================
+ Coverage   95.04%   95.06%   +0.01%     
==========================================
  Files          14       14              
  Lines         747      750       +3     
==========================================
+ Hits          710      713       +3     
  Misses         37       37              
Impacted Files Coverage Δ
sh_scrapy/extension.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c0c6f3...2c2ec56. Read the comment docs.

@elacuesta
Copy link
Member Author

itemadapter does not support py35, so the tests fail. I think it's a good opportunity to just remove support here too.

@elacuesta elacuesta changed the title Update itemadapter usage Update itemadapter, drop py35, test in py310 Mar 21, 2022
@elacuesta elacuesta changed the title Update itemadapter, drop py35, test in py310 Update dependencies, drop py35, test in py310, improve itemadapter performance Mar 21, 2022
@elacuesta elacuesta merged commit 026fec7 into master Mar 22, 2022
@elacuesta elacuesta deleted the itemadapter-is-item branch March 22, 2022 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants