Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

不同商圈之间有重复小区, communites表id唯一导致数据插入失败 #5

Open
wzrzt opened this issue Oct 12, 2018 · 0 comments

Comments

@wzrzt
Copy link

wzrzt commented Oct 12, 2018

由于不同商圈之间可能有重复的社区,但是communities表id不能重复,导致抓取第二个商圈时就运行失败了,所以需要做一下处理。仿照删除商圈id的做法,在小区信息插入数据库之前加入了一行删除已有社区id的代码。如下所示,Main.py 163行update_db函数。

def update_db(db_session, biz_circle, communities):
    """
    更新小区信息, 商圈信息
    """
    db_session.query(Community).filter(
        Community.biz_circle_id == biz_circle.id
    ).delete()

    for community_info in communities['list']:
        try:
            district_id = DISTRICT_MAP[community_info['district_name']]
            community = Community(biz_circle.city_id, district_id, biz_circle.id, community_info)
            
            db_session.query(Community).filter(
                Community.id == community.id
            ).delete()

            db_session.add(community)
        except Exception as e:
            # 返回的信息可能是错误的/不完整的, 如小区信息失效后返回的是不完整的信息
            # 如: http://sz.lianjia.com/xiaoqu/2414168277659446
            logging.error('错误: 小区 id: {}; 错误信息: {}'.format(community_info['community_id'], repr(e)))

    biz_circle.communities_count = communities['count']
    biz_circle.communities_updated_at = datetime.now()

    db_session.commit()
@wzrzt wzrzt changed the title communites表id唯一导致数据插入失败 不同商圈之间有重复小区, communites表id唯一导致数据插入失败 Oct 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant