Error when logging in to Wekan REST API when using Sandstorm Wekan

[xet7]

@kentonv

The problem is, on Standalone Wekan I can login to Wekan REST API using Wekan username and password, but on Sandstorm Wekan when I login with Google Auth, I get error with Sandstorm Webkey when I use Webkey as password. I don't know what other password I would use.

I tried this way:

1. On Wekan board create Webkey like in image by @ertanalytics
   https://github.com/wekan/wekan/wiki/Wekan-Sandstorm-cards-to-CSV-using-Python

Sure by using Python BeautifulSoup etc Wekan html and javascript could probably be parsed and used like in code above, but using REST API would be much nicer.

BTW @ertanalytics when I tried to use your code this way:

sudo apt install python-pip python-bs4 python-selenium python-pandas \
python-bson chromium-chromedriver chromium-browser-l10n

sudo pip install selenium --upgrade

sudo pip install splinter

And copied chromedriver from https://sites.google.com/a/chromium.org/chromedriver/home to my /usr/bin , I got this error, do you know solution to it?

$ python cards-to-csv.py 
Traceback (most recent call last):
  File "cards-to-csv.py", line 62, in <module>
    script = soup.find("script", attrs={'type' : 'text/inject-data'}).children.next()
AttributeError: 'NoneType' object has no attribute 'children'

2. I tried to login to Wekan REST API on Sandstorm:

curl https://USERNAME:[email protected]/users/login \
     -d "username=USERNAME&password=APIKEY"

3. I get curl response:

{
  "error": "internal-server-error",
  "reason": "Internal server error"
}

4. On Sandstorm Wekan board console I get this error:

Error: Unauthorized [Unauthorized]
    at Object.Authentication.checkUserId (server/authentication.js:6:21)
    at models/users.js:563:20
    at packages/simple_json-routes.js:98:9

5. But if I try some other password, I get:

{
  "error": 400,
  "reason": "Match failed"
}

Using Wekan REST API on Sandstorm could make easier:

• uploading data from scripts etc to Sandstorm Wekan
• have mobile apps that use Sandstorm Wekan
• integrations with Huginn
• incoming Webhooks, etc.

Is there some documentation how with Meteor app like Wekan it would be possible to access info outside of grain, for example so that when importing attachments from Trello, some Sandstorm-compatible way could be used to download Trello attachments to Wekan?

I have not yet tried could EtherCalc API also be used on Sandstorm.

[ocdtrekkie]

I noticed that the Python script pulls from Selenium. I get the impression he is actually instructing Selenium to open the full Wekan grain and operating it in an automated fashion. It's definitely vastly preferable to query the API.

This is probably the documentation you're looking for, if you haven't seen this already: https://docs.sandstorm.io/en/latest/developing/http-apis/

[ertanalytics]

It looks like it last worked on 9/1/2017 and after the credentials were put in it was showing the full json text data of the wekan board and the subsequent steps parsed that json data.

The error:

Traceback (most recent call last): File "cards-to-csv.py", line 62, in <module> script = soup.find("script", attrs={'type' : 'text/inject-data'}).children.next() AttributeError: 'NoneType' object has no attribute 'children'

Is related to the parser not finding what it is expecting to start the parsing. Any ideas to navigate to the json data should fix this problem. Please let me know and I will update my code. I am trying a few different paths to see if I can figure out a way to get back to the json data with card titles etc. I would like all the board data if possible for the export.

@ocdtrekkie - I am just instructing Selenium to open the URL. Once the page loads I grab the background data and immediately close the browser:

`browser2 = Browser('chrome')

browser2.visit(apiURLnoAuth) # Fill in the url

sleep(1) #Time in seconds

soup = BeautifulSoup(browser2.html,'html.parser')
browser2.quit()`

Once the data is assigned to the 'soup' variable, I am done with Selenium.

[xet7]

@ertanalytics

There is already a way to Import and Export Wekan board to and from JSON at Sandstorm Wekan, when you click board menu (hamburger icon, 3 lines) and click there Import or Export. But I don't know how to do this with API.

Trello import on Sandstorm Wekan does not import attachments yet, because Wekan does not have Sandstorm compatible access to outside of grain. Importing Trello attachments works on Standalone Wekan (like Docker etc) and then you can export Wekan JSON that includes attachments in JSON as Base64 encoded, and then import that Wekan JSON to Sandstorm with attachments in JSON.

[ertanalytics]

@xet7 - This appears to be broken as well in Sandstorm. The download starts, but fails 'Failed - Server problem' while download backup on Sandstorm continues to work.

On 9/1/2017 and earlier, visiting the API URL had the export data readily available and that was what the script I wrote was parsing.

[xet7]

@ertanalytics

What version of Wekan you have on Sandstorm? Have you updated to latest? I have 0.47.0

When updating, also update all your grains.

Please include your Sandstorm Wekan grain console logs so I can see exact errors.

[ertanalytics]

Version: 0.47.0

** SANDSTORM SUPERVISOR: Grain shutdown requested.
** SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns
** Starting Mongo...
2017-10-07T18:58:45.562+0000 I STORAGE Engine custom option: log=(prealloc=false,file_max=200KB)
about to fork child process, waiting until server is ready for connections.
forked process: 10
child process started successfully, parent exiting
** Starting Meteor...
{ [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' }
js-bson: Failed to load c++ bson extension, using pure JS version
Error: 59d92409a78c710200228a12 does not exist
at Object.Future.wait (/programs/server/node_modules/fibers/future.js:449:15)
at packages/meteor.js:213:24
at models/export.js:85:15
at Array.map (native)
at Exporter.build (models/export.js:80:60)
at models/export.js:32:62
at packages/simple_json-routes.js:98:9
- - - - -
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:200:22
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/collection/query.js:164:5
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:778:35
at Cursor.close (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:1009:5)
at Cursor.nextObject (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:778:17)
at commandHandler (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:741:14)
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/db.js:1903:9
at Server.Base._callHandler (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/base.js:453:41)
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/server.js:488:18
at [object Object].MongoReply.parseBody (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
at [object Object]. (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/server.js:446:20)
at emitOne (events.js:77:13)
at [object Object].emit (events.js:169:7)
at [object Object]. (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:207:13)
at emitTwo (events.js:87:13)
at [object Object].emit (events.js:172:7)

[xet7]

Is it only error with this Wekan grain?

Can you export some other Wekan grain?

[ertanalytics]

It is only happening to the one board.

Other wekan grains are exporting json correctly.

[xet7]

I added this info to Wekan wiki

Exporting Wekan board to JSON file with Bash script

1. On Wekan grain, get Webkey like this:

https://api-URL.SUBDOMAIN.sandcats.io#APIKEY

2. Modity URL, SUBDOMAIN and APIKEY to this Bash script that exports board to file directly:

curl https://Bearer:[email protected]/api/boards/sandstorm/export?authToken=#APIKEY > wekanboard.json

[kentonv]

It seems like you're successfully doing the things you need to do on Sandstorm to talk to a grain API. The errors are coming from Wekan, not Sandstorm.

With that said, it's likely that you'll want to change the way the Wekan API is authorized on Sandstorm. Sandstorm will automatically add headers to the API requests identifying the user -- the same headers it adds to all HTTP requests. https://docs.sandstorm.io/en/latest/developing/auth/#headers-that-an-app-receives

The X-Sandstorm-User-Id header in particular contains the Sandstorm identity ID, which matches the field services.sandstorm.id in the user record in Mongo. So you can figure out who the user is that way, instead of requiring a separate login step.

[ertanalytics]

As far as trying the Wekan API, I am not having any luck using the links through Sandstorm. The only thing that has worked was the export URL thus far. I am betting the directory structure inhibits the normal API request functions, however I am a bit green on API testing and might be missing something.

[ertanalytics]

I added a bounty to this issue to try to get links made for the API that also work in Sandstorm. Only the export link works and that is not part of the API:

https://github.com/wekan/wekan/wiki/REST-API

Bounty page:
https://www.bountysource.com/teams/wekan/issues

https://www.bountysource.com/issues/50166298-error-when-logging-in-to-wekan-rest-api-when-using-sandstorm-wekan

[xet7]

I think I could try to fix this by exposing /api at sandstorm-pkgdef.capnp file when I get some free time.

[ertanalytics]

Wish I had the setup and understanding to do these changes myself. Any idea on timeline?

[xet7]

@ertanalytics

No idea. Maybe in near or distant future.

[ertanalytics]

Following up on this one. I have reached out to xet7 to assist and he is having trouble with his sandstorm based building VM. If anyone has a working development environment to test the REST API for sandstorm version and build from it to get it to production please let me know, I would like to get this feature as soon as possible.

[xet7]

@ertanalytics

As you can see from these IRC chatlogs, only @kentonv is currently able to build Sandstorm from source, because he made changes how Sandstorm was built:
https://botbot.me/freenode/sandstorm/2018-02-11/?tz=Europe/Helsinki

I don't know are there pull request to documentation yet.

I also have not yet figured out all steps yet.

[xet7]

@ertanalytics

Probably these fixes to build script were recently merged:
sandstorm-io/sandstorm#3034

[xet7]

@ertanalytics

Add new issue to Sandstorm if you still have problems:
https://github.com/sandstorm-io/sandstorm/issues

[xet7]

@ertanalytics

And also add info to this issue if you get Sandstorm built, and how you did it.

[xet7]

@ertanalytics

I added issue to Sandstorm, I still have problems.
sandstorm-io/sandstorm#3036

[xet7]

Install instructions included at Sandstorm repo should now work also for you.
You can see changed merged pull requests about Docs fixes at https://github.com/sandstorm-io/sandstorm/pulls?q=is%3Apr+is%3Aclosed

So you can clone that repo and follow instructions. It should work OK.

[xet7]

Requirements:

1. Add cards - including specific labels, descriptions, checklists.
2. Move cards - from one list to another.
3. Remove members - from the entire board.
4. Change assignment - Assign certain members to certain cards or remove them.

[xet7]

@ertanalytics

Can you test:

• make backup of all grains
• upgrade to newest Wekan
• is there any difference?

[ertanalytics]

While I do backup nightly, I am not in a position where I can comfortably restore the exact same version of my environment if I upgrade and I fear change like most without thorough testing. There is already a way to test on Sandstorm's Oasis. If no changes were made to the API structure in wekan or sandstorm, I believe this bug will persist. I did a test below, however I am not super confident of my formatting for API requests in sandstorm apps other than the already working export API get request.

https://oasis.sandstorm.io/

I tested again, not that I really know how the format should go in Sandstorm for user addition, but I tried this and got a 'Bad Request' for user add while I was able to confirm export was still working:

import requests
import json

APIBegURL = 'https://Bearer:nS_0nDcc63vZ9QWpuNCkPK5BBMGLDiLxP-0JNhVKz79@api-a5587d9539ca906a2aa5e1eebc7dac89.oasis.sandstorm.io'
EndURL = '#nS_0nDcc63vZ9QWpuNCkPK5BBMGLDiLxP-0JNhVKz79'

Commands

#Confirm API still works for exporting entire board
exportCommand = '/api/boards/sandstorm/export?authToken='
APIRequest = APIBegURL + exportCommand + EndURL
response = requests.get(APIRequest)
print(response.text) ## Returns full export

#Bad Request Result for User Addition
usersCommand = '/api/boards/sandstorm/users?authToken='
APIRequest = APIBegURL + usersCommand + EndURL
response = requests.post(APIRequest,
data='',
json='{ "username": "tester", "password": "tester", "email": "[email protected]", "fromAdmin": "true" }'
)
print(response.text) ## Returns bad request

[ocdtrekkie]

@ertanalytics Did you test with 1.33 on Oasis or 1.34?

[ertanalytics]

Version 1.33.0~2018-08-16 @ocdtrekkie

[xet7]

@ertanalytics

Please test Wekan Sandstorm version v1.39, it has speedups, does it help enough with performance?

[ocdtrekkie]

Also, since you tested with the pre-SandstormApi change version, test out the API too and let me know if it behaves differently. I wanted to do some testing myself but I've been really busy of late and don't know how much time I have to devote to side projects at the moment.

[ertanalytics]

@xet7 , it is more about the API not working than performance. I have not noticed a performance increase with the update. My larger, more used board, takes about 23 seconds to respond when you click on the browser to view from being in the background. It is 20.8 MB board on Sandstorm.

@ocdtrekkie , I have updated to version 1.39 of Wekan on Sandstorm, updated Sandstorm with Dev channel and now the board export API above no longer works. Other API calls continue to not work. I get status OK response, but it just gives me the same response text no matter what API commands I put in:
`

<title>Wekan</title> <script type="text/inject-data">%7B%22fast-render-data%22%3A%7B%22collectionData%22%3A%7B%7D%2C%22subscriptions%22%3A%7B%22boards%22%3A%7B%22%5B%5D%22%3Atrue%7D%7D%7D%7D</script> <script type="text/javascript">__meteor_runtime_config__ = JSON.parse(decodeURIComponent("%7B%22meteorRelease%22%3A%22METEOR%401.6.0.1%22%2C%22meteorEnv%22%3A%7B%22NODE_ENV%22%3A%22production%22%2C%22TEST_METADATA%22%3A%22%7B%7D%22%7D%2C%22PUBLIC_SETTINGS%22%3A%7B%22sandstorm%22%3Atrue%7D%2C%22ROOT_URL%22%3A%22http%3A%2F%2F127.0.0.1%3A4000%22%2C%22ROOT_URL_PATH_PREFIX%22%3A%22%22%2C%22appId%22%3A%22dvyihgykyzec6y1dpg%22%2C%22SANDSTORM%22%3Atrue%2C%22accountsConfigCalled%22%3Atrue%2C%22autoupdateVersion%22%3A%2243fee1ee295407330ac727c00083754ef519cfaa%22%2C%22autoupdateVersionRefreshable%22%3A%22a20cc749bf1d8e6b7cee5304126efa2415be9d02%22%2C%22autoupdateVersionCordova%22%3A%22none%22%7D"))</script> <script type="text/javascript" src="/2355927353f489870710f283894fc0f2d5623705.js?meteor_js_resource=true"></script> `

[xet7]

@ertanalytics

Export using Wekan board hamburger menu / Export still works, so API is enabled on on Wekan side. I know this, because if at Standalone Wekan WITH_API=false, Exporting board at Wekan menu does not work, but gives similar HTML output like you have above. That settings WITH_API=true is at code here:
https://github.com/wekan/wekan/blob/devel/sandstorm-pkgdef.capnp#L240

I don't know what effect did addition of /api by @ocdtrekkie do:
https://github.com/wekan/wekan/blob/devel/sandstorm-pkgdef.capnp#L229

@kentonv

Does Sandstorm API still work? Or did some update disable it?

[xet7]

I only tested at sandstorm.io though. I'll test at something.sandcats.io next.

[xet7]

I tested something.sandcats.io , created webkey etc, it's the same. Export using API does not work. Export using Wekan menu does work. Something prevents calling API.

[ocdtrekkie]

Hm..... what if you remove the /api from your request URL, @xet7?

I'm wondering if setting apiPath to /api makes it add it to the URL, which would be /api/api. If this is true, we can change the apiPath to / to fix this.

[xet7]

@ocdtrekkie

/api/api and /api/api/api did not work yet in rest api. I try modifying sandstorm-pkgdef.capnp file.

[ocdtrekkie]

To clarify, I had meant to tell you to try the request WITHOUT the part of the URL, as I think my change is adding it.

…

Sent from my Windows 10 phone From: Lauri Ojansivu Sent: Tuesday, September 4, 2018 5:57 PM To: wekan/wekan Cc: Jacob Weisz; Mention Subject: Re: [wekan/wekan] Error when logging in to Wekan REST API when usingSandstorm Wekan (#1279) @ocdtrekkie /api/api and /api/api/api did not work. I try modifying js file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ertanalytics]

Will this commit be in an experimental version or on Oasis or somewhere I can test outside of my environment?

[ocdtrekkie]

My understanding is that the change xet7 made to un-break this is in the current version of Wekan, which I've approved to the app market. Note that before packages are approved, they can be found at https://apps.sandstorm.io/?experimental=true

[ertanalytics]

Is there a chance someone could test this functionality? I am getting hangs in the experimental link on both browser types. Do we know the normal testing procedure for sandstorm Wekan and what features are tested and if we can get this feature included?

[xet7]

Wekan is at official store, without ?experimental=true in url. I think there is no experimental currently. If official also does not work, it's more likely Sandstorm bug.

@kentonv What do you think?

[ocdtrekkie]

I still don't think there's any Sandstorm bugs with the API at present. I don't think the API has ever worked really correctly on the Sandstorm version.

@ertanalytics When there's no experimental app available, the experimental market stays on "loading". It's a known issue with the market because the experimental market is kinda a hack. sandstorm-io/sandstorm-app-market#98

[ertanalytics]

After running the same tests, I received the same bad response for the test response in the latest version on Oasis. Below is the testing method in Python 3.6.5 with requests package version number 2.18.4 on Oasis Wekan version 1.49.0-2018-09-17 tested 9/28/2018.

`import requests
import json

#https://api-ec9e7c51610a73ebf1e11001caf8b1dd.oasis.sandstorm.io#L92rf5G3B3rIsk8Z1USnnM-6ZjbRFugyWZWCaxP4iDt
#APIBegURL = 'https://Bearer:nS_0nDcc63vZ9QWpuNCkPK5BBMGLDiLxP-0JNhVKz79@api-a5587d9539ca906a2aa5e1eebc7dac89.oasis.sandstorm.io'
APIBegURL = 'https://Bearer:L92rf5G3B3rIsk8Z1USnnM-6ZjbRFugyWZWCaxP4iDt@api-ec9e7c51610a73ebf1e11001caf8b1dd.oasis.sandstorm.io'
EndURL = '#L92rf5G3B3rIsk8Z1USnnM-6ZjbRFugyWZWCaxP4iDt'
#Commands

#Confirm API still works for exporting entire board
exportCommand = '/api/boards/sandstorm/export?authToken='
APIRequest = APIBegURL + exportCommand + EndURL
response = requests.get(APIRequest)
print(response.text) ## Returns full export

#Bad Request Result for User Addition
usersCommand = '/api/boards/sandstorm/users?authToken='
APIRequest = APIBegURL + usersCommand + EndURL
response = requests.post(APIRequest,
data='',
json='{ "username": "tester", "password": "tester", "email": "[email protected]", "fromAdmin": "true" }'
)
print(response.text) ## Returns bad request`

[xet7]

I got API working in Wekan release I will make soon.

API works this way:

1. Make API key, and from that key copy API URL and API KEY to below. It saves Wekan board to file.

curl http://Bearer:[email protected]:6080/api/boards/sandstorm/export?authToken=#APIKEY > wekanboard.json

2. If later API key does not work, you need to remove it and make a new one.

[ertanalytics]

Verified the export does show data again. The other API features do not appear to work including the user addition example, but at least the export is working again with the API.