Replies: 15 comments
-
|
Ok so this will qualify as a big site :) I would ask for a repro but that's gonna be difficult. I suspect we don't have a mechanism (yet) for a local cache of all remote artifacts so that repeated builds do not need to redownload everything. Not very useful for you but helpful for repro/testing. I'm very interested in seeing the node count. This would be printed when you use |
Beta Was this translation helpful? Give feedback.
-
|
Yes sir, I'm on it, I will report back as soon as possible but in 10 hours from now I will be probably sleeping so it may take up to 18 hours Before creating this issue I did tried to test it couple of times to exclude any issues such as loss of network in the 8 hours time or bad connection, today morning I also gave it another run but it failed after 50 minutes of running however it didn't downloaded any new files and when it happened it "kind of finnished" but failed to connect to Drupal and drupal source was not visible in graphQL however on the bottom right page there was displayed "5 pages" (created from static .js pages) One more interesting fact which suprised me in regards to folder .cache all those GB are beign downloaded into: .cache\caches\gatsby-source-filesystem I was expecting images that are downloaded from drupal to be in: .cache\caches\gatsby-source-drupal Website may be actually bigger when it comes to total number of "nodes" in Drupal terminology there should be more than number of products as this website was also in 8 languages however only some products where translated in general there were around 40 000 products in 3 translations which would give around 120 000 nodes - but in Gatsby I do not generate any nodes programaticaly from the GraphQL as I was unable to fetch any data to display in GraphQL yet, in general I do plan display only nodes from one specific language. |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry about my reply, I was unable to test it with this command: node --max-old-space-size=14192 node_modules/.bin/gatsby build --verbose It gave following error: I modified my older command by adding --verbose to it and it somewhat moved forward but there was loss of connection during the night and it provided following error with some verbose information: |
Beta Was this translation helpful? Give feedback.
-
|
I can try to provide some context on using gatsby-source-drupal on a site this large:
Now for some more in-depth testing... The gatsby-source-drupal plugin scans the JSON:API of your Drupal site and uses that to pull down all the data. There are a number of potential things we can do to get data into your Gatsby site. These might not fix the underlying scaling problem, but might help us diagnose where the issues are (my guess is image processing, but we can test that):
Can you try a few of the above methods to see if you can get data pulled into gatsby develop and from there we can work on determining/fixing the biggest bottlenecks? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much for getting back to me I really appreciate that, I have been also following discussion that I have noticed on twitter today and I also come to some ideas (which are not very good most likely)
The general plan is to at least make this website work without any images I will do my best to test it out and report back to you. Hopefully this topic will help someone else in future as well. |
Beta Was this translation helpful? Give feedback.
-
|
I did confirm if you disable the file and media entities in the JSON:API Resource Overrides admin page it will not download those files (even if your content entities reference them). That step alone should get your local development site building. Let me know if that works to at least get the data without the images. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much! Just wanted to confirm after testing out with disabling the FILE and IMAGE STYLES it increased time for "npm start heavyload" (gatsby develop) from around 34603.075s (9.5 hours) to just 416.814s ( 7 minutes ) But no pages are yet generated dynamicaly from queries so time will defenitly increase after setting up the generation of product pages. (I will report back when I will be able to test it further) However I was trying to at least somehow access Absolute path for the missing images to try to somehow just print out them using I'm trying to think of a way how to make it possible to somehow display images without actually downloading them all. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Small case update, unfortunately I think I need to use Commerce API instead of just using standard JSON API (however) in standard JSON API I can use JSONAPI EXTRAS and this allows me to disable the "file--file" but when I use Commerce API it have conflict with JSONAPI EXTRAS and I no longer can disable the file--file from Drupal generated JSON file , any tips how to solve this? Still even if development server will load up in 10 minutes which would be great I would still need to find way to display images from somewhere. I have found this Gatsby issue which seems to be somehow related to my issue and it also seems that it was almost done? not sure if it would help me, but perhaps it could? unfortunately I do not know how to test it I guess I would need to wait for it to be part of the main gatsby branch. In addition to trying to find a way to deal with images and I was reading this Lullabot post not sure if it is up to date but I'm running out of ideas :( Link to module: Link to blogpost: |
Beta Was this translation helpful? Give feedback.
-
|
It looks like we have a few problems here. Downloading all of the images is taking a long time. The gatsby-source-drupal plugin is going to download these images as long as they are referenced in the JSON provided by JSON:API. The issue you linked is one potential solution to help with that. I will follow up on that issue to see what is blocking it from getting merged but it appears it does require some additional work. It looks like the commerce_api module will not work with jsonapi_extras. This is going to cause other problems down the line because the jsonapi_extras module is required if you want things like live preview or incremental builds to work. It might be worth following up in this thread to see what is causing to conflicts and if it is something commerce_api will be fixing in the future - https://www.drupal.org/project/commerce_api/issues/3121480 The idea of third party image hosting is a good one. There are options such as Cloudinary or you could host them on your Drupal site and potentially use the consumer_image_styles plugin. Other third party hosted CMS's such as Contentful and DatoCMS use this approach (which is why their build times are much faster). The images are never pulled down locally and the correct image is displayed at runtime based on the browser size. This would definitely speed up your builds but we would still need to figure out how to make sure the files were not downloaded (which is why the jsonapi_extras module would be helpful). |
Beta Was this translation helpful? Give feedback.
-
|
What I did in a similar situation was to rename |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for feedback. It is possible to prevent Gatsby from downloading images from "Commerce API" without changing the JSONAPI output (the load time is 30 minutes without images) by adding: (Unfortunately the href= for the images is no longer available so I can't even print out images from my website directly or trough 3rd party app. ) However it takes only 7 minutes to load gatsby develop if file--file resource is disabled as resource from "JSONAPI Extras" ( possibly it could be lowered even further using JSONAPI Boost (cache warmer that also doesn't work with Commerce API) Incremental builds seems like must have in my case, but I tought that Commerce API is successor of Commerce Cart API (which seemed logical to use the newer API on new project) The tricky part is to prevent gatsby downloading images (as option) when using plugin: gatsby-source-drupal but still have access to the "href" value in GraphQL for those images. I did tried to find part responsible for creating cache of images locally in the plugin, but I failed :( Hosting images on my Drupal website and serving them trough Consumers Image styles + Lazyloading them with height placeholder in place seems like best option - Thank you -- edit @ascorbic - thank you so much for this great idea I tried to change the name of resource file--file using JSONAPI Extras unfortunately while trying to connect with Gatsby it return Error 500 and on domain.com /jsonapi (this error may be caused by my hosting.. but after reverting to original file--file error no longer appears Could you also by any chance provide more informations about how you wrote this custom resolver? |
Beta Was this translation helpful? Give feedback.
-
|
As far as the 500 error. JSON:API does have some caching (especially if you are pulling the data through JSON:API as an anonymous user). Just to be sure, it might be worth making sure you clear the Drupal cache and run |
Beta Was this translation helpful? Give feedback.
-
|
One other option you might consider is trying There are definitely some improvements needed to gatsby-source-drupal to work around some of these image scaling issues. |
Beta Was this translation helpful? Give feedback.
-
|
Something i did for custom entity with a multi-valued image field is to write a custom base field like so: $fields['banner_images'] = BaseFieldDefinition::create('string')
->setLabel(t('Banners JSON'))
->setDescription(t('The JSON version of banner image files'))
->setComputed(TRUE)
->setClass(ComputedBanners::class);And the class: <?php
namespace Drupal\rlh_hotel\Plugin\Field;
use Drupal\Core\Field\FieldItemList;
use Drupal\Core\TypedData\ComputedItemListTrait;
use Drupal\Component\Serialization\Json;
class ComputedBanners extends FieldItemList {
use ComputedItemListTrait;
/**
* {@inheritdoc}
*/
protected function computeValue() {
$entity = $this->getEntity();
$banner_images = [];
foreach ($entity->get('field_banner_images') as $banner_image) {
$file = $banner_image->entity;
$image = [
'url' => $file->url(),
'alt' => $banner_image->alt,
'width' => $banner_image->width,
'height' => $banner_image->height,
];
$banner_images[] = $image;
}
$encoded = !empty($banner_images) ? Json::encode($banner_images) : null;
$this->list[0] = $this->createItem(0, $encoded);
}
}Now this new field is available in the json api extras settings, and i set enhancer for that field to "JSON Field". The end result is i get a url + attributes to link to on the gatsby side. It doesn't get to use all the gatsby image goods cause the images stay stored on drupal site but if you have a ton of images, this might be an option for you. If you insist on downloading files, the ideas above are pretty good. I'd urge you to disable all the links you don't need in the json api settings and particularly the |
Beta Was this translation helpful? Give feedback.
-
|
@apmsooner the changes to skip certain downloads has been merged now. Should help you a bit, hopefully :) |
Beta Was this translation helpful? Give feedback.
-
Summary
Hello,
I have website built on Headless Drupal 8 with around 30 000 products (+10 000 out of stock) I started with building frontend on Gatsby without connecting to my Drupal 8 backend when I tried to connect it using gatsby-source-drupal plugin a lot of different problems started to show up.
Each product have 5 images so in total there should be around 200 000 images
Basic example
Javascript out of heap error - when reaching 2 GB (solved) by adding new command in package.json
"heavyload": "node --max-old-space-size=14192 ./node_modules/gatsby/dist/bin/gatsby.js develop"
and running: npm run heavyload
Fetching - info Starting to fetch data from Drupal
success Fetch data from Drupal - 2623.108s - take around 43 minutes
info Downloading remote files from Drupal <- and now the real problem begins, in order to run the Gatsby Develop our script starts to download all files into cache files (around 50GB) unfortunately if it fails while downloading those (50 GB) files and I want to run Gatsby develop again, I need to download all 50 GB from scratch (it already happened twice to me) and it takes around 8 hours to download 80 gb of files.
After running the Gatsby Develop for past 20 hours I was unable to finish this process even once while connected to my Drupal website I'm attaching log of failed attempt
Motivation
I'm not sure what I'm doing wrong, but I don't know honestly how to deal with it, I want to connect my Gatsby website with my Drupal 8 website, but if it takes around 8 hours to connect and after 8 hours I get error and have to try from scratch it, I really don't know what to do.
Perhaps it is my API fault, or Gatsby Drupal plugin have missing configuration, I can share my API but I wouldn't want to make it public.
Beta Was this translation helpful? Give feedback.
All reactions