ghub-request results in a 400 error w/ error message: "Problems parsing JSON

[whompyjaw]

edit: I opened emacs 29.1 compiled with gnutls on the system that worked for 29.1.50 (SLES15.4), and it also worked. System 1 is SLES15.4 and system 2 is SLES12.5)
edit2: This is the error message reported back from the github server (i assume)
json-read-from-string("{"message":"Problems parsing JSON","documentation_url":"https://docs.github.com/[email protected]...")... So maybe whatever you're doing for the query in ghub-request doesn't work for 3.7?
I have emacs 29.1 compiled w/ gnutls (my work compiled this) and emacs 29.1.50 that I compiled myself without gnutls. Forge pull works on the latter, but not the former. The original error was this, which was misleading...
Messages buffer

Opening TLS connection to ‘github.host.com’...
Opening TLS connection with ‘gnutls-cli --x509cafile /etc/ssl/ca-bundle.pem -p 443 github.host.com’...done
Opening TLS connection to ‘github.host.com’...done
json-read-string: End of file while parsing JSON
Package tls is deprecated

Though I am using a github enterprise instance I don't think this issue applies: #174 because I am able to make the request on my other emacs version. And we are using host/api/v3 not host/v3

• On the system with emacs 29.1, I can make a successful curl request to the host: curl -H "Authorization: Bearer token" -H "Content-Type: application/json" https://github.host.com/api/graphql returns a very large json object

global git config

user.name=username
[email protected]
core.editor=emacs
core.fsmonitor=true
init.defaultbranch=main
github.host.user=username
github.host.com/api/v3.user=username
github.github.host.com/api/v3.user=username

If you look in the traces, the 29.1.50 uses url-http, while the 29.1 uses gnutls (Messages above)

Here is backtrace for emacs 29.1 w/ gnutls

json-read-string fails
 ghub--handle-response((:error (error http 400)) #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "github.host.com" :portspec nil :filename "/api/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x1ee89718dc5bc9b4>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--retrieve("{\"query\":\"query ($name: String!,$owner: String!) {..." #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "github.host.com" :portspec nil :filename "/api/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x1ee89718dc5bc9b4>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  #f(compiled-function (arg1 arg2 &rest rest) "LONG STRING" #<bytecode -0x1db2c53fdb2e9da>)("POST" "/graphql" nil :payload ((query . "query ($name: String!,$owner: String!) {\n  reposit...") (variables (owner . "Repo") (name . "repo"))) :headers nil :silent nil :username nil :auth forge :host "github.host.com/api/v3" :forge nil :callback nil :errorback nil :extra nil :value nil)
  apply(#f(compiled-function (arg1 arg2 &rest rest) "LONG STRING" #<bytecode -0x1db2c53fdb2e9da>) ("POST" "/graphql" nil :payload ((query . "query ($name: String!,$owner: String!) {\n  reposit...") (variables (owner . "Repo") (name . "repo"))) :headers nil :silent nil :username nil :auth forge :host "github.host.com/api/v3" :forge nil :callback nil :errorback nil :extra nil :value nil))
* ghub-request("POST" "/graphql" nil :payload ((query . "query ($name: String!,$owner: String!) {\n  reposit...") (variables (owner . "Repo") (name . "repo"))) :headers nil :silent nil :username nil :auth forge :host "github.host.com/api/v3" :forge nil :callback nil :errorback nil :extra nil :value nil)
  ghub-graphql((query (repository [(owner $owner String!) (name $name String!)] id)) ((owner . "repo") (name . "repo")) :username nil :auth forge :host "github.host.com/api/v3")
  ghub--repository-id("repo" "repo" :username nil :auth forge :host "github.host.com/api/v3")
  ghub-repository-id("repo" "repo" :host "github.host.com/api/v3" :auth forge :forge github)
  #f(compiled-function (class host owner name &optional stub) "Return (OUR-ID . THEIR-ID) of the specified repository.\nIf optional STUB is non-nil, then the IDs are not guaranteed to\nbe unique.  Otherwise this method has to make an API request to\nretrieve THEIR-ID, the repository's ID on the forge.  In that\ncase OUR-ID derives from THEIR-ID and is unique across all\nforges and hosts." #<bytecode -0x3f83b23ca765525>)(forge-github-repository "github.host.com" "Repo" "repo" nil)
  apply(#f(compiled-function (class host owner name &optional stub) "Return (OUR-ID . THEIR-ID) of the specified repository.\nIf optional STUB is non-nil, then the IDs are not guaranteed to\nbe unique.  Otherwise this method has to make an API request to\nretrieve THEIR-ID, the repository's ID on the forge.  In that\ncase OUR-ID derives from THEIR-ID and is unique across all\nforges and hosts." #<bytecode -0x3f83b23ca765525>) forge-github-repository ("github.host.com" "Repo" "repo" nil))
  forge--repository-ids(forge-github-repository "github.host.com" "Repo" "repo" nil)
  #f(compiled-function (&rest rest) "((HOST OWNER NAME) &optional REMOTE DEMAND)\n\nReturn the repository identified by HOST, OWNER and NAME.\nSee `forge-alist' for valid Git hosts." #<bytecode -0x1a7bc2b2576a458b>)(("github.host.com" "Repo" "repo") "origin" :insert!)
  apply(#f(compiled-function (&rest rest) "((HOST OWNER NAME) &optional REMOTE DEMAND)\n\nReturn the repository identified by HOST, OWNER and NAME.\nSee `forge-alist' for valid Git hosts." #<bytecode -0x1a7bc2b2576a458b>) ("github.host.com" "Repo" "repo") ("origin" :insert!))
  forge-get-repository(("github.host.com" "Repo" "repo") "origin" :insert!)
  #f(compiled-function (url &optional remote demand) "Return the repository at URL." #<bytecode 0x1b0aaa358e357020>)("[email protected]:Repo/repo.git" "origin" :insert!)
  apply(#f(compiled-function (url &optional remote demand) "Return the repository at URL." #<bytecode 0x1b0aaa358e357020>) "[email protected]:Repo/repo.git" ("origin" :insert!))
  forge-get-repository("[email protected]:Repo/repo.git" "origin" :insert!)
  #f(compiled-function (demand &optional remote) "Return the current forge repository.\n\nFirst check if `forge-buffer-repository', or if that is nil, then\nthe repository for `forge-buffer-topic', satisfies DEMAND.  If so,\nthen return that repository.\n\nOtherwise return the repository for `default-directory', if that\nexists and satisfies DEMAND.  If that fails too, then return nil\nor signal an error, depending on DEMAND." #<bytecode -0x675e57a41862c05>)(:insert!)
  apply(#f(compiled-function (demand &optional remote) "Return the current forge repository.\n\nFirst check if `forge-buffer-repository', or if that is nil, then\nthe repository for `forge-buffer-topic', satisfies DEMAND.  If so,\nthen return that repository.\n\nOtherwise return the repository for `default-directory', if that\nexists and satisfies DEMAND.  If that fails too, then return nil\nor signal an error, depending on DEMAND." #<bytecode -0x675e57a41862c05>) :insert! nil)
  forge-get-repository(:insert!)
  forge-pull(nil nil t)
  funcall-interactively(forge-pull nil nil t)
  call-interactively(forge-pull record nil)
  command-execute(forge-pull record)
  execute-extended-command(nil "forge-pull" nil)
  funcall-interactively(execute-extended-command nil "forge-pull" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Here is backtrace for emacs 29.1.50* without backtrace

* json-read-string()
  json-read-object()
  json-read()
  json-read-from-string("{\"data\":{\"repository\":{\"name\":\"dragonr\",\"id\":\"MDEw...")
  ghub--read-json-payload(200)
  ghub--handle-response-payload(#s(ghub--graphql-req :url #s(url :type "https" :user nil :password nil :host "github.host.com" :portspec nil :filename "/api/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x1ee8ad6edea4d534>) :handler ghub--graphql-handle-response :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (data) #<bytecode -0x3b34af6cb721d5f>) :errorback nil :value nil :extra nil :query (query (repository [(owner $owner String!) (name $name String!)] name id createdAt updatedAt nameWithOwner (parent nameWithOwner) description homepageUrl (defaultBranchRef name) isArchived isFork isLocked isMirror isPrivate hasIssuesEnabled hasWikiEnabled (licenseInfo name) (stargazers totalCount) (watchers totalCount) (assignableUsers [(:edges t)] id login name) (issues [(:edges t) (:singular issue number) (orderBy (... ...))] number id state stateReason isReadByViewer (author login) title createdAt updatedAt closedAt locked (milestone id) body (assignees [(:edges t)] id) (comments [(:edges t)] id databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)) (labels [(:edges t) (:singular label id)] id name color description) (milestones [(:edges t) (:singular milestone id)] id number title createdAt updatedAt dueOn closedAt description) (pullRequests [(:edges t) (:singular pullRequest number) (orderBy (... ...))] number id state isReadByViewer (author login) title createdAt updatedAt closedAt mergedAt isDraft locked maintainerCanModify isCrossRepository (milestone id) body (baseRef name (repository nameWithOwner)) baseRefOid (headRef name (repository (owner login) nameWithOwner)) headRefOid (assignees [(:edges t)] id) (reviewRequests [(:edges t)] (requestedReviewer "... on User { id }\n")) (comments [(:edges t)] id databaseId (author login) createdAt updatedAt body) (labels [(:edges t)] id)))) :query-str "query ($name: String!,$owner: String!) {\n  reposit..." :variables ((owner . "Repo") (name . "repo")) :until ((issues-until . "2024-03-08T18:38:29Z") (pullRequests-until . "2024-03-14T16:13:04Z")) :buffer #<buffer xilinx-prims> :pages 1 :paginate nil))
  ghub--graphql-handle-response((:peer ("I REMOVED THIS") ("I REMOVED THIS") (:version 3 :serial-number "39:72:44:3a:f9:22:b7:51:d7:d3:6c:10:dd:31:35..." :issuer "C=GB,ST=Place Place,L=Salford,O=Comod..." :valid-from "2019-03-12" :valid-to "2028-12-31" :subject "C=US,ST=New Jersey,L=Jersey City,O=The USERT..." :public-key-algorithm "RSA" :certificate-security-level "High" :signature-algorithm "RSA-SHA384" :public-key-id "sha1:c8:95:13:68:01:97:28:0a:2c:55:c3:fc:d3:..." :certificate-id "sha1:d8:9e:3b:d4:3d:5d:90:9b:47:a1:89:77:aa:..." :pem "-----BEGIN CERTIFICATE-----\nMIIFgTCCBGmgAwIB...") (:version 3 :serial-number "01" :issuer "C=GB,ST=Place Place,L=Salford,O=Comod..." :valid-from "2004-01-01" :valid-to "2028-12-31" :subject "C=GB,ST=Place Place,L=Salford,O=Comod..." :public-key-algorithm "RSA" :certificate-security-level "Medium" :signature-algorithm "RSA-SHA1" :public-key-id "sha1:c4:30:28:c5:d3:e3:08:0c:10:44:8b:2c:77:..." :certificate-id "sha1:d1:eb:23:a4:6d:17:d6:8f:d9:25:64:c2:f1:..." :pem "-----BEGIN CERTIFICATE-----\nMIIEMjCCAxqgAwIB...")) :certificate (:version 3 :serial-number "33:72:6c:98:28:3b:99:d8:41:5d:02:df:8d:5a:fd..." :issuer "C=GB,ST=Place Place,L=Salford,O=Secti..." :valid-from "2023-06-06" :valid-to "2024-06-05" :subject "C=US,ST=STATE,O=HOST INCORPORATED,C..." :public-key-algorithm "RSA" :certificate-security-level "Medium" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:ba:c8:8f:9d:d8:de:69:d6:9e:41:15:81:d8:..." :certificate-id "sha1:00:3a:54:7f:2e:ee:8e:29:4a:55:f6:76:bb:..." :pem "-----BEGIN CERTIFICATE-----\nMIIG2TCCBcGgAwIB...") :key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-128-GCM" :mac "AEAD" :encrypt-then-mac nil :safe-renegotiation t)) #s(ghub--graphql-req :url #s(url :type "https" :user nil :password nil :host "github.host.com" :portspec nil :filename "/api/graphql" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x1ee8ad6edea4d534>) :handler ghub--graphql-handle-response :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (data) #<bytecode -0x3b34af6cb721d5f>) :errorback nil :value nil :extra nil :query (query (repository [(owner $owner String!) (name $name String!)] name id createdAt updatedAt nameWithOwner (parent nameWithOwner) description homepageUrl (defaultBranchRef name) isArchived isFork isLocked isMirror isPrivate hasIssuesEnabled hasWikiEnabled (licenseInfo name) (stargazers totalCount) (watchers totalCount) (assignableUsers [(:edges t)] id login name) (issues [(:edges t) (:singular issue number) (orderBy ...)] number id state stateReason isReadByViewer (author login) title createdAt updatedAt closedAt locked (milestone id) body (assignees [...] id) (comments [...] id databaseId (author login) createdAt updatedAt body) (labels [...] id)) (labels [(:edges t) (:singular label id)] id name color description) (milestones [(:edges t) (:singular milestone id)] id number title createdAt updatedAt dueOn closedAt description) (pullRequests [(:edges t) (:singular pullRequest number) (orderBy ...)] number id state isReadByViewer (author login) title createdAt updatedAt closedAt mergedAt isDraft locked maintainerCanModify isCrossRepository (milestone id) body (baseRef name (repository nameWithOwner)) baseRefOid (headRef name (repository ... nameWithOwner)) headRefOid (assignees [...] id) (reviewRequests [...] (requestedReviewer "... on User { id }\n")) (comments [...] id databaseId (author login) createdAt updatedAt body) (labels [...] id)))) :query-str "query ($name: String!,$owner: String!) {\n  r..." :variables ((owner . "Repo") (name . "repo")) :until ((issues-until . "2024-03-08T18:38:29Z") (pullRequests-until . "2024-03-14T16:13:04Z")) :buffer #<buffer xilinx-prims> :pages 1 :paginate nil))
  url-http-activate-callback()
  url-http-chunked-encoding-after-change-function(62607 62688 81)
  url-http-generic-filter(#<process github.host.com> "u\17\363\304k\256\322\350b\245\344]\222\321\363\337:\237\231\3\n\377\306\340$\346\261X\363\312\3615N\371\243\344S\302\272\210q\260\217\213\354\272\330\321\0231...")

My init.el section

(use-package magit
  :custom
  (magit-refresh-verbose t)
  (magit-auto-revert-mode t))
(use-package forge
  :after magit
  :config
  (add-to-list 'forge-alist
	   '("github.host.com"
	     "github.host.com/api/v3"
	     "github.host.com"
	     forge-github-repository)))

Any thoughts? I will continue to debug this.

[whompyjaw]

Ok so trying to do the request manually on my own i am able to get a response back:

   "query": "query { repository(owner: \"Org\", name: \"repo\") { pullRequests(states: OPEN, first: 100) { edges { node { title, url } } } } }"
}' https://github.host.com/api/graphql

So I think there is something wrong with the syntax of the request. Possibly it's the fact that my work is sadly using GitHub 3.7 and forge supports only newer graphql syntax?

[whompyjaw]

I don't really see a syntax query difference between the 3.12 and 3.7 github enterprise graphql, so I wonder if this is an actual issue with ghub package... Looking at the ghub log, seems there were recent changes made to the query processing...
magit/ghub@1326315

[whompyjaw]

I've tried checking out older versions of ghub (3.4.0, 3.5.6, and 3.6.0) and they all result in the same error... I am really not sure what else to try here... Or how else I can debug this. Any help would be great appreciated it.

[tarsius]

Any help would be great appreciated it.

I saw your requests for help, but have been very busy over the last few hours and so wasn't able to look into this yet.

[whompyjaw]

No worries! I don't expect anything within 24 hours. I appreciate you getting back. Hopefully you didn't take my comments as rushing. I was just adding more info as I was thinking of different things to try, haha. :)

[tarsius]

There appear to be two problems:

1. Ghub tries to parse errordata as JSON and fails. That might be a bug in Ghub or it may be that Github changed how they encode such data.
2. Github returns a 400 error. That does not necessarily mean (and likely doesn't) that the requested resource does not exist. They also use this error code if authentication failed, in order to avoid revealing the existence of some resource to someone who is not authorized. Maybe you try to authenticate as the wrong user. Maybe your token has expired.

[tarsius]

I've pushed 9f152cf, which makes it unnecessary to define a new class. If you have already done so, remove all repositories that use it from the database and only then remove the class.

I cannot really test this myself and rely on you to do it.

Edit: You have to add two entries to ghub-insecure-hosts, see updated https://github.com/magit/forge/wiki/Tips-and-Tricks#accessing-unsecured-forges.

[tarsius]

I have 2 situations where ghub-request is not in the stacktrace

That's because the request is made asynchronously. A callback is involved. There is no straight trace from making the request to receiving the response.

[whompyjaw]

I didn't know that it is possible to setup an unsecured github instance. (You are the first Forge user trying to connect to one. I've heard about more than one unsecured Gitlab instance from Forge users already.)

I don't believe it's insecure. My work's github instance is at https and chrome says it's secure, so I assume it is.

After your update, I still get the same error:

Opening TLS connection to ‘github.company.com’...
Opening TLS connection with ‘gnutls-cli --x509cafile /etc/ssl/ca-bundle.pem -p 443 github.company.com’...done
Opening TLS connection to ‘github.company.com’...done
json-read-string: Bad string format: 7
OR
json-read-string: End of file while parsing JSON
Package tls is deprecated

So, I am thinking it may be the emacs I am using. It is my work's emacs and it was compiled with gnutls (which I assume would work), and it was not compiled with json-support (libjannsson)... It worked on a different emacs, before. So I might try to compile my own emacs and let you know.

But to compare the 2 emacs':
The one that doesn't work was compiled with this:

--with-x-toolkit=yes --with-xpm=ifavailable --with-jpeg=ifavailable --with-gif=ifavailable --with-gnutls=ifavailable --with-mailutils --prefix=/software/emacs/29.1

Where as, the other emacs (one in zypper package manager for SLES15 SP4) that worked is compiled with this:

--disable-build-details --with-pop --without-hesiod --with-gameuser=:games --with-kerberos --with-kerberos5 --with-file-notification=inotify --with-modules --enable-autodepend --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --localstatedir=/var --sharedstatedir=/var/lib --libexecdir=/usr/lib --enable-locallisppath=/usr/share/emacs/27.2/site-lisp:/usr/share/emacs/site-lisp --with-x --with-xim --with-sound --with-xpm --with-jpeg --with-tiff --with-gif --with-png --with-rsvg --with-dbus --with-xft --without-gpm --with-x-toolkit=gtk3 --with-toolkit-scroll-bars --x-includes=/usr/include --x-libraries=/usr/lib64 --with-libotf --with-m17n-flt --with-cairo --with-xwidgets --build=x86_64-suse-linux --with-dumping=pdumper ’CFLAGS=-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -D_GNU_SOURCE -DGDK_DISABLE_DEPRECATION_WARNINGS -DGLIB_DISABLE_DEPRECATION_WARNINGS -pipe -Wno-pointer-sign -Wno-unused-variable -Wno-unused-label -fno-optimize-sibling-calls -fno-PIE -DSYSTEM_PURESIZE_EXTRA=55000   -DSITELOAD_PURESIZE_EXTRA=10000  -DPDMP_BASE=’\’’"emacs-gtk"’\’’’ ’LDFLAGS=-Wl,-no-pie -Wl,-O2’

Which does not have TLS compiled with it... So what's going on here? Am I suppose to be using tls or not? I get a deprecation warning for tls in emacs 29.1 compiled with gnutls, and that doesn't work for Forge (but org-jira works fine), but i have an emacs not compiled with tls and it works and your wiki message says:

Initially only secured forges (those using tls) were supported

I am all kinds of confused...

I just want one system that can use org-jira and magit-forge... I am so close lol.

[tarsius]

I don't believe it's insecure. My work's github instance is at https and chrome says it's secure, so I assume it is.

Well the following put me on the wrong path:

As an aside, can you explain when or how forge (or show the code) that decides when to make an http request or an https (SSL/TLS) request? Because I have 2 different emacs making url-http without ghub-request and one that is making it.

On the plus side, for people who actually have to connect to unsecured instances, it is a bit easier now thanks to the changes I have made.

[whompyjaw]

That was my mistake. I was throwing darts in the beginning, so may have led you astray. But, yes, on the plus side, glad you could add something to help others with unsecured. That's great you added that.

[tarsius]

With the "unsecured instance" thing out of the way, I can start looking at the actual problem you are having.

What Emacs version are you currently using?

[tarsius]

(I'll deal with magit/ghub#166 first. Addressing that might make it easier to deal with this.)

[whompyjaw]

Sorry for my delay. I thought I responded to this. I am now currently using 29.3 that I compiled myself. It works as expected now. I can use forge and org-jira. I am not sure if it could be the --with-json requirement (as i had that in my other 29.1.50 version that i compiled)

--prefix=/custom/path/emacs/29.3 --with-native-compilation=aot --with-tree-sitter --with-json --with-gnutls --with-x-toolkit=lucid --with-imagemagick --with-mailutils --with-xpm --with-jpeg --with-gif --with-png --with-rsvg --with-tiff --with-sound=no

The other emacs I was using was 29.1 compiled by my work (i mentioned it above)

--with-x-toolkit=yes --with-xpm=ifavailable --with-jpeg=ifavailable --with-gif=ifavailable --with-gnutls=ifavailable --with-mailutils --prefix=/software/emacs/29.1