Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Face detection #341

Open
whyboris opened this issue Jan 19, 2020 · 15 comments · May be fixed by #342
Open

Face detection #341

whyboris opened this issue Jan 19, 2020 · 15 comments · May be fixed by #342
Assignees
Labels
discussion Thoughts and ideas
Milestone

Comments

@whyboris
Copy link
Owner

whyboris commented Jan 19, 2020

It seems like it may be relatively simple to perform face detection (placing a bounding box across any face found in an image). This process may be super fast: apparently faster than realtime for video (which means stupid-fast for a few screenshots).

It may be simple enough to simply detect any faces within a filmstrip and extract them into a new faces array which can be viewed separately somehow (would need UI work -- unsure what it would look like yet).

Possibly the best solution: https://github.com/tehnokv/picojs

Another alternative is: https://github.com/eduardolundgren/tracking.js but it hasn't been updated since 2018 😕

Another alternative is: https://github.com/justadudewhohacks/face-api.js/ (see Tiny Face Detector), but it requires TensorFlow and I don't want some heavy dependencies for this minor feature 😅

Someday in far future, we could get very advanced with face recognition too: https://github.com/justadudewhohacks/face-api.js/

🤔 🤷‍♂ 😁

@whyboris whyboris added the discussion Thoughts and ideas label Jan 19, 2020
@whyboris whyboris added this to the Ideas milestone Jan 19, 2020
@whyboris
Copy link
Owner Author

Update:

  • Tried picojs and it's not a good fit: it has many false-positives (e.g. finds faces in elbows)
  • Tried face-api and I'm really liking it. Assuming including TensorFlow and other sub-dependencies isn't going to make the app too heavy, I just might include it. We'll see 🤷‍♂

I have a script that seems to generate an image strip with faces found in all the extracted screenshots. I'm thinking create a new view that looks like the filmstrip view, but it will only show faces 🤔

I'm open for ideas about how to display the found faces and what to do with them. 👂

@cal2195
Copy link
Collaborator

cal2195 commented Jan 21, 2020

Well this sounds very exciting! 😄

Two thoughts about what you could do with the data:

  • If you can find the same face in multiple videos, this would be a very cool improvement to the similar video recommendations (Right click -> Show more videos with this actor)

  • Secondly, I see you can also detect expressions - this would be a very cool tag to add, if the user enabled it. (eg. tag videos with happy, surprised etc.)

I like this direction a lot! 🎉

@whyboris
Copy link
Owner Author

whyboris commented Jan 23, 2020

I have a repository for the internal functionality: https://github.com/whyboris/extract-faces-node

You can take any thumbnail strip from your library and run extract faces.

One caveat is you have to set the number of screenshots there are in the filmstrip inside the script:
https://github.com/whyboris/extract-faces-node/blob/master/src/pipeline.ts#L8

🚀 I'll get a branch of VHA ready to see how well it plays with Electron 😅

@whyboris
Copy link
Owner Author

Currently running into trouble when trying to install canvas 😓 -- so far no-go ...
Electron post-install script errors out not being able to find a pre-built binary(?)

@whyboris
Copy link
Owner Author

Problem solved 🎉 😂 whyboris/extract-faces-node#1 -- was able to remove canvas 😁
Attempting again with VHA 🤞

@whyboris
Copy link
Owner Author

Seems to work well 😎 #342
App compiles to 125MB 👌
Finds and extracts faces while in dev but not yet from the built app -- I'll investigate 🔍 😁 🔎

@whyboris whyboris linked a pull request Jan 24, 2020 that will close this issue
@whyboris whyboris modified the milestones: Ideas, Likely Jan 24, 2020
@whyboris whyboris self-assigned this Jan 24, 2020
@frez777
Copy link
Contributor

frez777 commented Jan 25, 2020

Pretty cool idea!

Well, even the idea of adding an actor section itself is very useful and nice feature.

@whyboris
Copy link
Owner Author

@frez777 -- btw -- have you clicked on the "Auto Tags"? It will automatically find all the 2-word strings that appear in your files (generally actor names if they appear in the file names).

I'll likely expand this feature more -- thank you for the request 👍

@whyboris whyboris added the help wanted Extra attention is needed label Jan 25, 2020
@whyboris
Copy link
Owner Author

help wanted : if you have experience and time, could you see how to get the built application to extract images?

Branch face-detection (see PR: #342) -- when npm start the app extracts faces 👍 but when npm run electron:windows the app doesn't extract faces 👎

@whyboris
Copy link
Owner Author

Things I've tried:

  • Make sure the Node is the same version as what Electron uses ✅ -- seems like it may be required for rebuilding native dependencies step

@whyboris
Copy link
Owner Author

whyboris commented Jan 25, 2020

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 PROBLEM SOLVED 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

Turns out the weights folder needs to be copied correctly once the app installs (e.g. inside Program Files/Video Hub App 2/... 👍

I should have a build-able branch ready later today (see #342)

@whyboris whyboris removed the help wanted Extra attention is needed label Jan 25, 2020
@whyboris
Copy link
Owner Author

Current intention for the future features, not all will be in #342:

  1. is to extract every face for every filmstrip and store it in a new folder (with the same file name as the hash as the file).
  2. add a 'faces' view that will simply show a filmstrip with all the faces found
  3. create another 'auto-generated tag' view that shows only the two-word tags (hopefully mostly names), but next to each it will have a single photo
  4. each of the photos can be replaced by dragging-dropping an image file.
  5. this new view will also (like the auto-generated tags) allow to add more names to the list
  6. the view will behave like the auto-generated tag view -- clicking will open all the videos that contain that 'name'

These are the initial features -- should be pretty easy to implement. Once it's set and released, I will consider others.

Before that's done -- I'm open to more suggestions / recommendation for change, etc.

Part of the reason I'm aiming for those 6 things is that they are easy from the code point of view (re-using same functionality a lot). We'll see how it goes.

@whyboris whyboris pinned this issue Jan 27, 2020
@whyboris
Copy link
Owner Author

CONTINUED DISCUSSION OF THIS FEATURE HERE: #342

@whyboris whyboris reopened this Jan 31, 2020
@whyboris
Copy link
Owner Author

Feature is further postponed until I release version 3.0.0 - see #456 🙆‍♂️

@whyboris whyboris mentioned this issue Nov 4, 2020
1 task
@whyboris
Copy link
Owner Author

This feature is not abandoned. I've been wanting to work on it for 3 years now, but after moving to another state, continuing a full-time developer job, and spending time on fixing up the house, I've just not had time to resume this feature yet 😓

But in 2024 I intend to find time for it 🤞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Thoughts and ideas
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants