Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New YTMusic Cookie Format? #195

Open
jrlebbs opened this issue Sep 28, 2024 · 23 comments · May be fixed by #203
Open

New YTMusic Cookie Format? #195

jrlebbs opened this issue Sep 28, 2024 · 23 comments · May be fixed by #203

Comments

@jrlebbs
Copy link

jrlebbs commented Sep 28, 2024

Please check the FAQ before submitting a bug report.

Describe the bug
YouTube Music stopped authenticating as a source about 3 days ago. Logged in following the guide and the cookie now starts with "HSID" instead of "SID"

To Reproduce
Steps to reproduce the behavior:

  1. Follow the steps on the config guide: https://foxxmd.github.io/multi-scrobbler/docs/configuration/#youtube-music

Expected behavior
The cookie should auth YTMusic.

Logs
If possible reproduce the issue with debug logging ON

2024-09-27T23:27:29-06:00 debug : [Notifiers] [Ntfy - Multi-Scrobbler] Pushed notification.
2024-09-27T23:27:28-06:00 error : [Sources] [Ytmusic - YTMusic] Cannot start polling because source is not authenticated correctly.
2024-09-27T23:27:28-06:00 info : [App] Server started at http://localhost:9078
2024-09-27T23:27:27-06:00 warn : [Sources] ytmusic (YTMusic) source auth failed.
2024-09-27T23:27:26-06:00 debug : [Sources] Checking ytmusic (YTMusic) source auth...
2024-09-27T23:27:26-06:00 info : [Sources] ytmusic (YTMusic) source initialized
2024-09-27T23:27:26-06:00 debug : [Sources] Attempting ytmusic (YTMusic) initialization...
2024-09-27T23:27:26-06:00 debug : [Sources] (YTMusic) Constructing ytmusic source
2024-09-27T23:27:26-06:00 debug : [Sources] listenbrainz has 'configureAs: client' so will skip adding as a source
2024-09-27T23:27:26-06:00 debug : [Sources] lastfm has 'configureAs: client' so will skip adding as a source
2024-09-27T23:27:26-06:00 info : [Scrobblers] listenbrainz (brainz) client auth OK
2024-09-27T23:27:25-06:00 debug : [Scrobblers] Checking listenbrainz (brainz) client auth...
2024-09-27T23:27:25-06:00 info : [Scrobblers] listenbrainz (brainz) client initialized
2024-09-27T23:27:25-06:00 debug : [Scrobblers] Attempting listenbrainz (brainz) initialization...
2024-09-27T23:27:25-06:00 debug : [Scrobblers] Constructing listenbrainz (brainz) client...
2024-09-27T23:27:25-06:00 info : [Scrobblers] lastfm (myLastFm) client auth OK
2024-09-27T23:27:25-06:00 info : [API - Lastfm - myLastFm] Client authorized for user jrlebbs
2024-09-27T23:27:25-06:00 debug : [Scrobblers] Checking lastfm (myLastFm) client auth...
2024-09-27T23:27:25-06:00 info : [Scrobblers] lastfm (myLastFm) client initialized
2024-09-27T23:27:25-06:00 debug : [Scrobblers] Attempting lastfm (myLastFm) initialization...
2024-09-27T23:27:25-06:00 debug : [Scrobblers] Constructing lastfm (myLastFm) client...
2024-09-27T23:27:25-06:00 info : [Notifiers] [Ntfy - Multi-Scrobbler] Initialized. Found Ntfy server
.

Versions (please complete the following information):
Provide version information for any related sources/clients.

  • multi-scrobbler: unsure, installed about a year ago and not upgraded since. request help on how to find, if required
    -- running on windows

Additional context
I have tried several ways, reboots, etc. to get this to work. I suspect the cookie format has changed and I'm happy to provide more cookie details, just unsure of what's secure or not to be careful with.

Happy to help in any way!

@FoxxMD
Copy link
Owner

FoxxMD commented Sep 30, 2024

You should update your instance of Multi-Scrobbler...the last year of updates has really improved MS's stability and added a ton of features including knowing what version you are on.

However I am able to reproduce on current MS versions as well.

[2024-09-30 12:19:45.718 -0400] ERROR  : [App] [Sources] [Ytmusic - MyYTM] Authentication test failed!
Error: Authentication test failed!
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:235:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at YTMusicSource.initialize (CWD/src/backend/common/AbstractComponent.ts:56:13)
    at ScrobbleSources.addSource (CWD/src/backend/sources/ScrobbleSources.ts:596:18)
    at ScrobbleSources.buildSourcesFromConfig (CWD/src/backend/sources/ScrobbleSources.ts:497:25)
    at <anonymous> (CWD/src/backend/index.ts:115:9)
caused by: Error: Could not send the specified request to browse. Status code: 404
    at m.<anonymous> (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:55514)
    at p (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:46814)
    at Generator.<anonymous> (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:48160)
    at Generator.next (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:47243)
    at a (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:53888)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The endpoint is for music.youtube.com/youtubei/v1 with post data to the playlist id FEmusic_history. It's unclear if it's this specific endpoint of the playlist ID that is causing the 404.

MS uses another library to facilitate this communication so it's not something I can immediately fix or investigate but it looks like other projects relying on the endpoint are having issues as well. It may be that Youtube Music hanged their undocumented API endpoints or there is an internal issue that could be resolved.

@bockbilbo
Copy link

Coming from #199. Here are my logs, exactly the same as @FoxxMD:

[2024-10-08 22:25:03.884 -0700] DEBUG  : [App] [Sources] (Youtube Music) Constructing ytmusic source
[2024-10-08 22:25:03.885 -0700] DEBUG  : [App] [Sources] [Ytmusic - Youtube Music] Attempting to initialize...
[2024-10-08 22:25:03.885 -0700] VERBOSE: [App] [Sources] [Ytmusic - Youtube Music] Building required data init succeeded => Read initial credentials from config
[2024-10-08 22:25:03.886 -0700] DEBUG  : [App] [Sources] [Ytmusic - Youtube Music] Connection check was not required.
[2024-10-08 22:25:03.948 -0700] ERROR  : [App] [Sources] [Ytmusic - Youtube Music] Authentication test failed!
Error: Authentication test failed!
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:235:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at YTMusicSource.initialize (CWD/src/backend/common/AbstractComponent.ts:56:13)
    at ScrobbleSources.addSource (CWD/src/backend/sources/ScrobbleSources.ts:596:18)
    at ScrobbleSources.buildSourcesFromConfig (CWD/src/backend/sources/ScrobbleSources.ts:497:25)
    at <anonymous> (CWD/src/backend/index.ts:115:9)
caused by: Error: Could not send the specified request to browse. Status code: 404
    at m.<anonymous> (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:55514)
    at p (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:46814)
    at Generator.<anonymous> (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:48160)
    at Generator.next (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:47243)
    at a (CWD/node_modules/youtube-music-ts-api/build/exports.js:3:53888)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-10-08 22:25:03.953 -0700] INFO   : [App] [Sources] [Ytmusic - Youtube Music] Fully Initialized!

@FoxxMD
Copy link
Owner

FoxxMD commented Oct 9, 2024

Mostly talking to myself here but I've confirmed that the liked music endpoint still works when using oauth authentication with the original ytmusicapi python library. Using browser authentication, which is what yt-music-ts-api (MS) uses, is what is failing when trying to hit the endpoint.

FoxxMD added a commit that referenced this issue Oct 9, 2024
* Fixes #195 missing (404) history endpoint which only appear when using cookies
* Might fix #158 due to using a different authentication method
@FoxxMD FoxxMD linked a pull request Oct 9, 2024 that will close this issue
4 tasks
FoxxMD added a commit that referenced this issue Oct 9, 2024
* Fixes #195 missing (404) history endpoint which only appear when using cookies
* Might fix #158 due to using a different authentication method
@FoxxMD
Copy link
Owner

FoxxMD commented Oct 9, 2024

@jrlebbs @bockbilbo please try out the new YTM auth in #203 and let me know if this fixes things for you -- foxxmd/multi-scrobbler:pr-203 and check out the Usage section in the PR comment.

@bockbilbo
Copy link

@FoxxMD the new pr is not working, I am getting an exception for a missing package:

Node.js v20.11.1
multi-scrobbler is starting!
node:internal/process/esm_loader:34
      internalBinding('errors').triggerUncaughtException(
                                ^
Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'typescript-json-schema' imported from /app/src/backend/utils/SchemaUtils.ts
    at packageResolve (node:internal/modules/esm/resolve:853:9)
    at moduleResolve (node:internal/modules/esm/resolve:910:20)
    at defaultResolve (node:internal/modules/esm/resolve:1130:11)
    at nextResolve (node:internal/modules/esm/hooks:865:28)
    at resolveBase (file:///app/node_modules/tsx/dist/esm/index.mjs?1728527467923:2:3212)
    at resolveDirectory (file:///app/node_modules/tsx/dist/esm/index.mjs?1728527467923:2:3584)
    at resolveTsPaths (file:///app/node_modules/tsx/dist/esm/index.mjs?1728527467923:2:4073)
    at resolve (file:///app/node_modules/tsx/dist/esm/index.mjs?1728527467923:2:4447)
    at nextResolve (node:internal/modules/esm/hooks:865:28)
    at Hooks.resolve (node:internal/modules/esm/hooks:303:30) {
  code: 'ERR_MODULE_NOT_FOUND'
}
Node.js v20.11.1

FoxxMD added a commit that referenced this issue Oct 10, 2024
* Fixes #195 missing (404) history endpoint which only appear when using cookies
* Might fix #158 due to using a different authentication method
FoxxMD added a commit that referenced this issue Oct 10, 2024
* Fixes #195 missing (404) history endpoint which only appear when using cookies
* Might fix #158 due to using a different authentication method
@FoxxMD
Copy link
Owner

FoxxMD commented Oct 10, 2024

Please pull the latest PR image and see if that works, thanks

@bockbilbo
Copy link

I have tried pr-203 and pr-205. The logs from my previous message are for pr-203. pr-205 was not even booting. What latest PR image do you refer to?

FoxxMD added a commit that referenced this issue Oct 11, 2024
* Fixes #195 missing (404) history endpoint which only appear when using cookies
* Might fix #158 due to using a different authentication method
@FoxxMD
Copy link
Owner

FoxxMD commented Oct 11, 2024

Please try re-pulling foxxmd/multi-scrobbler:pr-203 as it has new fixes/code from today, thanks. After starting MS please confirm it shows version as pr203-72d79fcafa497b0fdbabcffbfb164b1f012457fd

@bockbilbo
Copy link

Thanks @FoxxMD . I have tried again, and have confirmed the version. The tool is prompting me a code to Re-authenticate into Youtube Music, but redirects to an internal website "/api/source/auth?name=Youtube%20Music&type=ytmusic" that returns "Internal Server Error".

Here are the logs from the app in Docker:

[2024-10-11 17:54:13.499 -0700] WARN   : [App] [Sources] [Ytmusic - Youtube Music] Polling is already stopped!
Error: You must be signed in to perform this operation.
    at Session.signOut (file:///app/node_modules/youtubei.js/dist/src/core/Session.js:157:19)
    at YTMusicSource.clearCredentials (/app/src/backend/sources/YTMusicSource.ts:128:32)
    at YTMusicSource.reauthenticate (/app/src/backend/sources/YTMusicSource.ts:122:20)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at <anonymous> (/app/src/backend/server/auth.ts:54:17)
    at async wrappedMiddleware (/app/node_modules/@awaitjs/express/index.js:116:7)

My guess is that i should have been redirected to a youtube page to enter the code to authenticate myself. Do i need to enter some kind of Youtube developer API token in the multi-scrobbler configs?

@bockbilbo
Copy link

bockbilbo commented Oct 12, 2024

Actually.. I was able to connect to youtube music and scrobble music played from their website! Scrobbling is not working though when I play music from the Youtube Music app. Is that behavior expected?

I found this log on the tool:

[2024-10-11 17:49:51.317 -0700] ERROR  : [App] [Sources] [Ytmusic - Youtube Music] Authentication test failed!
Error: Authentication test failed!
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:235:31)
    at YTMusicSource.initialize (CWD/src/backend/common/AbstractComponent.ts:56:13)
    at <anonymous> (CWD/src/backend/index.ts:124:17)
caused by: Error: Sign in with the code 'XXX-XXX-XXXX' using the authentication link on the dashboard or https://www.google.com/device
    at YTMusicSource.doAuthentication (CWD/src/backend/sources/YTMusicSource.ts:139:27)
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:229:27)
    at YTMusicSource.initialize (CWD/src/backend/common/AbstractComponent.ts:56:13)
    at <anonymous> (CWD/src/backend/index.ts:124:17)

I used the google.com/device url to make it work! I guess that the redirect url link in the app needs review?

@jrlebbs
Copy link
Author

jrlebbs commented Oct 13, 2024

Ok, I've got 0.8.5 installed. When I run npm start (before I get 203 merged), this happens:

MINGW64 ~/multi-scrobbler-0.8.5
$ npm start

[email protected] start
NODE_ENV=production tsx src/backend/index.ts

'NODE_ENV' is not recognized as an internal or external command,
operable program or batch file.

@MyNameHand
Copy link

MyNameHand commented Oct 13, 2024

Actually.. I was able to connect to youtube music and scrobble music played from their website! Scrobbling is not working though when I play music from the Youtube Music app. Is that behavior expected?

I found this log on the tool:

[2024-10-11 17:49:51.317 -0700] ERROR  : [App] [Sources] [Ytmusic - Youtube Music] Authentication test failed!
Error: Authentication test failed!
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:235:31)
    at YTMusicSource.initialize (CWD/src/backend/common/AbstractComponent.ts:56:13)
    at <anonymous> (CWD/src/backend/index.ts:124:17)
caused by: Error: Sign in with the code 'XXX-XXX-XXXX' using the authentication link on the dashboard or https://www.google.com/device
    at YTMusicSource.doAuthentication (CWD/src/backend/sources/YTMusicSource.ts:139:27)
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:229:27)
    at YTMusicSource.initialize (CWD/src/backend/common/AbstractComponent.ts:56:13)
    at <anonymous> (CWD/src/backend/index.ts:124:17)

I used the google.com/device url to make it work! I guess that the redirect url link in the app needs review?

Yea so I was able to do the same, and it does work with mobile app for me. But I noticed that it "discovered" 20 tracks, which were the last ones I had played before setting up the scrobbler. If I try to play those, it doesn't scrobble, but anything not on that list scrobbles. However, they get added to that list and don't scrobble anymore if I replay the song. Is this intended an behavior?

firefox_vSWnLWinBB

@bockbilbo
Copy link

@MyNameHand I have experienced the same behavior. After my first run, multi-scrobbler got the 20 Youtube Music plays before setting up the scrobbler, but then did not update despite refreshing every 10 seconds. A day later, it refreshed some 10 songs I played from my google speaker, but then stopped detecting more after that.

@FoxxMD I wonder if there is a problem with the code part that retrieves and refreshes the cache from Youtube Music. Or could it be a problem with the google url used to get the data?

@FoxxMD
Copy link
Owner

FoxxMD commented Oct 14, 2024

Yeah I need to do some more due diligence on how the data returned looks compared to the previous cookie method. The current changes in #203 were just to see if oauth would work (At all). Will look into the behavior ya'll have described this week.

@FoxxMD
Copy link
Owner

FoxxMD commented Oct 15, 2024

Looks like YTM made other changes other than blocking cookie-based retrival of the history endpoint. The way it returned data it used to include all instances of played songs (without a timestamp) but they were at least ordered based on when they were played IE:

* Song B (today) <--- last played song
* Song C (today)
* Song A (today)
* Song D (yesterday)
* ...

We play Song A again...

* Song A (today) <--- last played song
* Song B (today)
* Song C (today)
* Song A (today) <--- same song but played earlier in the day
* Song D (yesterday)
* ...

Now, it's removing the earlier play and putting it at the front of the list:

* Song B (today) <--- last played song
* Song C (today)
* Song A (today)
* Song D (yesterday)
* ...

We play Song A again...

* Song A (today) <--- last played song
* Song B (today)
* Song C (today) <--- song A is completely removed
* Song D (yesterday)
* ...

Since YTM doesn't provide timestamp and the api doesn't allow tracking the player in realtime the logic used to determine new plays is dependent on the history order being "valid" with some heuristics. The new behavior is causing it to throw away the remove-and-bump differences which only affects tracks that are played again within YTM's history window.

FoxxMD added a commit that referenced this issue Oct 15, 2024
Should fix new way YTM orders recent history #195
@FoxxMD
Copy link
Owner

FoxxMD commented Oct 15, 2024

Please update to the latest image of foxxmd/multi-scrobbler:pr-203. This should fix the ordering issue, I hope.

@bockbilbo
Copy link

It works!!! I have just played song A, then song B, then song A again from the browser, and the three were recorded! :) I will continue to test in the next few days and get back if there is any problem. Thanks a lot @FoxxMD

@bockbilbo
Copy link

Alright... so.. after the test above, i tried using Youtube Music on my phone, and the song I played was scrobbled. However, a song I played a few days when testing was also scrobbled with the current timestamp.

@jrlebbs
Copy link
Author

jrlebbs commented Oct 16, 2024

@FoxxMD, I've got pr-203 pulled into docker, but when I try to run, I get the error below. Any thoughts on what I can do to fix?

docker run -v "$(pwd)/config:/config" -p 9078:9078 foxxmd/multi-scrobbler:pr-203
exec /init: exec format error

@FoxxMD
Copy link
Owner

FoxxMD commented Oct 16, 2024

I cannot reproduce that error and nothing has changed about the docker image with this PR. Are you using portainer for container management? If so make sure command and Entrypoint are set to default

image

@jrlebbs
Copy link
Author

jrlebbs commented Oct 16, 2024

No, I'm not using portainer. Just pulled the image and ran it with the commands.

@jrlebbs
Copy link
Author

jrlebbs commented Oct 17, 2024

I'm also fine just to wait until 203 is merged with main and pull the new main version, since it seems like other people's testing is working.

@FoxxMD
Copy link
Owner

FoxxMD commented Oct 17, 2024

@bockbilbo I've added a new config option that will help with troubleshooting what tracks are scrobbled. Please pull the latest pr-203 image and use a file/aio config for youtube music and add "logDiff": true" to the options, example:

[
  {
    "name": "MyYTMusic",
    "enable": true,
    "clients": [],
    "data": {
    },
    "options": {
      "logDiff": true
    }
  }
]

This will make MS log to DEBUG the list of tracked history and how it was modified each time MS decides to scrobble new tracks. Please see if you can reproduce the unexpected behavior and then describe in as much detail what you did along with logs from before/during/after the behavior occurred. You may also need to enable MS to log to debug for docker output/file by setting the env LOG_LEVEL=debug in your docker run command or compose file.

The logged history diff should look something like

[2024-10-17 15:29:01.474 +0000] DEBUG  : [App] [Sources] [Ytmusic - MyYTM] Changes from last seen list:
    1. (tuhe1CpHRxY) KNOWER - I’m The President --- undefined => Moved -  Originally at 6
2. (Mtg8V6Xa2nc) Vulfpeck - Romanian Drinking Song --- Schvitz => Moved -  Originally at 1
3. (rxbCaiyYSXM) Nightmares On Wax - You Wish --- In A Space Outta Sound => Moved -  Originally at 2
4. (tMt_YXr90AM) Gorillaz - O Green World --- undefined => Moved -  Originally at 3
5. (p5_C4J-MlO8) The Avalanches - Running Red Lights (feat. Rivers Cuomo & Pink Siifu) --- We Will Always Love You => Moved -  Originally at 4
6. (ovWFEOqrkBI) Gorillaz - Baby Queen --- undefined => Moved -  Originally at 5
7. (QooxlzmTjhs) KNOWER - Knower Forever --- KNOWER FOREVER
8. (nuI_0bnEwog) Jim Hall - Theme from "Black Orpheus" --- Wonder Boy
9. (0gOX4W2PNGM) Paul Desmond - A Taste of Honey --- Glad To Be Unhappy
10. (V-98pKLmGYA) Mulgrew Miller - Yardbird Suite --- Mulgrew Miller Solo
11. (cUuD8BUmvvU) Billie Holiday - Solitude --- Solitude
12. (oiB29EX0fZk) Billy Taylor - Joy Spring --- Joy Spring
13. (RKMrkMwLD28) Tony Bennett - Spring In Manhattan --- The Many Moods Of Tony
14. (mJWHc3ElGxE) Tinariwen - Amoss Idjraw --- undefined
15. (qYIx0T1O6-c) Frank Morgan - Up Jumped Spring --- Mood Indigo
16. (DDy5nifjbJ0) Oscar Peterson / Benny Green - Here's That Rainy Day --- Oscar And Benny
17. (jRFE3Akp2to) Bill Evans Trio - Spring Is Here --- Portrait In Jazz
18. (6IaXNmmthUM) Martin Landström - Lazy Art --- Lazy Art
19. (AUcuvqYRtxI) Sonny Stitt - Easy Does It --- Verve Jazz Masters 50: Sonny Stitt
20. (zYRI0vKKF5Y) Miles Davis / John Coltrane - How Am I to Know --- The Giants Miles Davis & John Coltrane
21. ```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants