forked from ibm-cloud-docs/containers
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathcontainer_troubleshoot.html
1138 lines (1121 loc) · 94.1 KB
/
container_troubleshoot.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html><html lang="en-us" xml:lang="en-us">
<head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="UTF-8">
<meta name="dcterms.date" content="2017-11-10">
<meta name="dcterms.rights" content="© Copyright IBM Corporation 2014, 2018">
<meta name="description" content="If you have problems or questions when you are using IBM Cloud Container Service, you can get help by searching for information or by asking questions through a forum. You can also open a support ticket.">
<meta name="keywords" content="troubleshoot, support, IBM Containers, containers, debug, fail, troubleshooting, cli, quota, images, Delivery pipeline, container, liberty, image, single containers, container groups, auto-recovery, group instances fail, group instances crash, delete, network_degraded, docker compose, data storage, volumes">
<meta name="geo.country" content="ZZ">
<script>
digitalData = {
page: {
pageInfo: {
language: "en-us",
version: "v18",
ibm: {
country: "ZZ",
type: "CT701"
}
}
}
};
</script><!-- Licensed Materials - Property of IBM -->
<!-- US Government Users Restricted Rights -->
<!-- Use, duplication or disclosure restricted by -->
<!-- GSA ADP Schedule Contract with IBM Corp. -->
<link rel="stylesheet" type="text/css" href="./ibmdita.css">
<title>Troubleshooting single and scalable containers</title>
</head>
<body><main role="main"><div><article class="nested0" role="article" aria-labelledby="d81670e6" id="container_troubleshoot"><h1 class="topictitle1" id="d81670e6">Troubleshooting <span class="ph">single and scalable containers</span> (Deprecated)</h1>
<div class="body taskbody"><div class="abstract"><div class="shortdesc"> If you have problems or questions when you are using IBM® Cloud Container Service, you can get help by searching
for information or by asking questions through a forum. You can also open a support ticket. </div>
<div class="p"><div class="note attention"><span class="attentiontitle">Attention:</span> Single and scalable containers are deprecated. Support
will continue in IBM Cloud Public until 5 December 2017 and in IBM Cloud Local and Dedicated until
20 June 2018. Start using Kubernetes clusters today to deploy secure, highly available apps. <a href="https://www.ibm.com/blogs/bluemix/2017/07/deprecation-single-scalable-group-container-service-bluemix-public/" rel="external" target="_blank" title="(Opens in a new tab or window)">Learn more about Kubernetes and how to migrate your
apps</a>.</div>
</div>
<div class="p">In this
page:<div class="lines"> • <a href="container_troubleshoot.html#ts_debug_containers" title="Review the options that you have to debug your containers and find the root causes for failures.">Debugging containers and reviewing logs</a><br>
• <a href="container_troubleshoot.html#ts_ov_cli">Troubleshooting the CLI setup</a><br>
• <a href="container_troubleshoot.html#ts_ov_orgs">Troubleshooting quota</a><br>
• <a href="container_troubleshoot.html#ts_ov_images">Troubleshooting images</a><br>
• <a href="container_troubleshoot.html#ts_ov_containers">Troubleshooting containers</a><br>
• <a href="container_troubleshoot.html#ts_ov_compose">Troubleshooting Docker Compose</a><br>
• <a href="container_troubleshoot.html#ts_ov_volumes">Troubleshooting data storage</a><br>
• <a href="container_troubleshoot.html#ts_known_issues" title="Learn about the known issues that occur in IBM Containers.">Known issues</a><br>
• <a href="container_troubleshoot.html#ts_getting_help" title="Where do you start troubleshooting a container?">Getting help and support for IBM Cloud Container Service</a></div>
</div>
</div>
</div>
<article class="topic task nested1" role="article" aria-labelledby="d81670e127" lang="en-us" id="ts_debug_containers"><h2 class="topictitle2" id="d81670e127">Debugging containers and reviewing logs</h2>
<div class="body taskbody"><p class="shortdesc">Review the options that you have to debug your containers and find the root causes for
failures. </p>
<ol class="steps"><li class="step stepexpand"><span class="cmd">List your containers and review the current status.</span> <ul><li><span class="keyword option">Single
container</span><pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> ps -a</code></pre>
</li>
<li><span class="keyword option">Container
group</span><pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> groups</code></pre>
</li>
</ul>
</li>
<li class="step stepexpand">Optional: <span class="cmd">See <a href="container_cli_reference_cfic.html#container_cli_reference_cfic_states" title="You can view the current container state by running the bx ic inspect command and locating the Status field. The container state can give you additional information about the availability of the container instances and problems that might have occurred.">what
each status means</a> and review tips to prevent a container from failing. </span></li>
<li class="step stepexpand"><span class="cmd">If your container failed or crashed, inspect your containers to find the root cause for the
failure. </span> The error message can be found in the <span class="ph uicontrol">Failure</span> section of your response. <ul><li><span class="keyword option">Single
container</span><pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> inspect <var class="keyword varname">CONTAINER</var></code></pre>
</li>
<li><span class="keyword option">Container group</span><ol type="a"><li>Inspect the container
group.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-inspect <var class="keyword varname">GROUP</var></code></pre>
</li>
<li>For further information, list the instances of a container group and inspect each instance.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-instances <var class="keyword varname">GROUP</var></code></pre>
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> inspect <var class="keyword varname">GROUP_INSTANCE</var></code></pre>
</li>
</ol>
</li>
</ul>
</li>
<li class="step stepexpand"><span class="cmd">If the app exited with a failure, retrieve the logs for the app. </span> <div class="steps note"><span class="notetitle">Note:</span> This command can be used only if the app writes logs to the standard STDOUT and STDERR output
streams. Writing to these standard output streams allows you to view container logs even if the
container shuts down or crashes.</div>
<ul><li><span class="keyword option">Single
container</span><pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> logs <var class="keyword varname">CONTAINER</var></code></pre>
</li>
<li><span class="keyword option">Container group</span><ol type="a"><li>List the container instances of a container
group.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-instances <var class="keyword varname">GROUP</var></code></pre>
</li>
<li>Retrieve the logs for each instance.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> logs <var class="keyword varname">GROUP_INSTANCE</var></code></pre>
</li>
</ol>
</li>
</ul>
</li>
<li class="step stepexpand"><span class="cmd">If you need to access logs of a container that is no longer available, such as an instance of a
container group that has been re-created with autorecovery, use Kibana to view your logs.</span> <p><samp class="ph codeph"><span class="ph filepath">https://logmet.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>/</span></samp></p>
<p>Logs are available for up to 7 days in Kibana.</p>
</li>
<li class="step stepexpand"><span class="cmd">If the logs were saved into a file inside the container, you must log into the container to
review it. </span> <div class="note important"><span class="importanttitle">Important:</span> You can only log into a running container. If your container crashed and
cannot be started again, you might not be able to access the logs of your container. Consider to
change your app to write to the standard STDOUT and STDERR output streams so that you can access the
log files even if the container is not accessible anymore.</div>
<ol type="a"><li>If your container is not running, start your container.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> start <var class="keyword varname">CONTAINER</var></code></pre>
</li>
<li>Log into your running container.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> exec -it <var class="keyword varname">CONTAINER</var> bash</code></pre>
</li>
<li>Navigate to your log file and review the logs. </li>
</ol>
</li>
<li class="step stepexpand"><span class="cmd">If the logs were saved to a volume and your container cannot be started anymore, create a new
container to access the volume. </span> <ol type="a"><li>Create a new container that is mounted to the volume where the log files are stored.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> run --name logcontainer -m 64 --volume <var class="keyword varname">MOUNT_PATH_TO_VOLUME</var> registry.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>/ibmliberty</code></pre>
</li>
<li>Log into the running container.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> exec -it logcontainer bash</code></pre>
</li>
<li>Navigate to the volume mount path and review your log files. </li>
</ol>
</li>
<li class="step stepexpand"><span class="cmd">If you need further support with debugging your containers, open a <a href="container_troubleshoot.html#ts_getting_help" rel="external" target="_blank" title="(Opens in a new tab or window)">support ticket</a>. </span></li>
</ol>
</div>
<aside role="complementary" aria-labelledby="d81670e127"></aside></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e391" lang="en-us" id="ts_ov_cli"><h2 class="topictitle2" id="d81670e391">Troubleshooting the CLI setup</h2>
<div class="body conbody"><p class="shortdesc"></p>
<p></p>
</div>
<aside role="complementary" aria-labelledby="d81670e391"></aside><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e452" lang="en-us" id="ts_cf_ic_rpc"><h3 class="topictitle3" id="d81670e452">Method error <samp class="ph codeph">(rpc)</samp> during init</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you try to run <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span>
init</samp>, you get one of the following messages.<ul><li><samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> rpc: can't find method
CliRpcCmd.IsLoggedIn</samp></li>
<li><samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> rpc: can't find method
CliRpcCmd.IsMinCliVersion</samp></li>
</ul>
</div><div class="section tsCauses">The
installed version of the Cloud Foundry is an earlier version than
what is supported
for <span class="keyword">IBM
Cloud Container Service</span>.</div><div class="section tsResolve"><ol><li>Run <samp class="ph codeph">cf -v</samp> to see which version of the Cloud
Foundry CLI is installed.</li>
<li>If the installed version is earlier than <span class="ph">6.14.0</span>, <a href="https://github.com/cloudfoundry/cli/releases" rel="external" target="_blank" title="(Opens in a new tab or window)">update your Cloud
Foundry CLI</a>.</li>
</ol>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e557" lang="en-us" id="ts_cf_ic_registry_error"><h3 class="topictitle3" id="d81670e557">Registry authentication error received during <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> init</samp></h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you run <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span>
init</samp>, you see one of the following authentication error messages.<div class="p"><ul><li><pre class="codeblock"><code>Could not authenticate with IBM Containers registry at registry.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>
****exit status 1
****time="2015-07-31T08:58:32-04:00" level=fatal msg="Post http:///var/run/docker.sock/v1.18/auth: dial
unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?”</code></pre>
</li>
<li><pre class="codeblock"><code>** Retrieving client certificates from IBM Containers
Authentication failure. Check your username and password
Invalid token format. Please generate new token.</code></pre>
</li>
</ul>
</div>
</div><div class="section tsCauses">Either one of the <span class="ph"><samp class="ph codeph">bx ic</samp></span> prerequisites
is not met or <span class="keyword">IBM
Cloud Container Service</span> is not
available.</div><div class="section tsResolve"><ol><li>Verify that the Docker daemon is running. If it is not, start it and run <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> init</samp> again.</li>
<li>Verify that you are logged in to the Cloud Foundry CLI by running <samp class="ph codeph">bx login</samp>.
Then, log in to the plug-in again by running <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> init</samp>.</li>
<li>Verify the availability of <span class="keyword">IBM
Cloud Container Service</span>.
<a href="https://developer.ibm.com/bluemix/support/#status" rel="external" target="_blank" title="(Opens in a new tab or window)">Check
the <span class="keyword">Cloud</span> status page for
information.</a></li>
</ol>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e688" lang="en-us" id="ts_sudo"><h3 class="topictitle3" id="d81670e688">Sudo is required with <span class="ph"><samp class="ph codeph">bx ic</samp></span>
commands</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms"> When you run <span class="ph"><samp class="ph codeph">bx ic</samp></span> commands, you
are required to use the <samp class="ph codeph">sudo</samp> prefix with all of your commands.</div><div class="section tsCauses">By default, the <span class="ph"><samp class="ph codeph">bx ic</samp></span> commands must be
run with root authentication and therefore, might require the prefix <samp class="ph codeph">sudo</samp>,
depending on your operating system.</div><div class="section tsResolve">To run <samp class="ph codeph">docker</samp> and <span class="ph"><samp class="ph codeph">bx ic</samp></span> commands without <samp class="ph codeph">sudo</samp>, run
the following command, then, if you are on Linux, you must log out then log back in again.
<pre class="codeblock"><code>sudo usermod -a -G docker <var class="keyword varname"><CURRENT_USER></var></code></pre>
</div></div>
</article></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e780" lang="en-us" id="ts_ov_orgs"><h2 class="topictitle2" id="d81670e780">Troubleshooting quota </h2>
<div class="body conbody"><p class="shortdesc"></p>
<p></p>
</div>
<aside role="complementary" aria-labelledby="d81670e780"></aside><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e841" lang="en-us" id="ts_quota_space"><h3 class="topictitle3" id="d81670e841">A container cannot be created because of insufficient org quota</h3>
<div class="body tsBody"><div class="section tsSymptoms">When you create a container, you receive a message that you do not have sufficient quota
in your
space.<pre class="pre msgblock"><code>BXNUI0110E: Could not allocate IBM Containers resources. Insufficient ORG quota to allocate
to new SPACE. Cannot exceed your organization '<var class="keyword varname">organization</var>' floating IPs. Use IBM
<span class="keyword">Cloud</span> Manage Organizations to
adjust quota allocation.</code></pre>
</div><div class="section tsCauses">This message might be displayed when you try to create containers in a space that does not
have enough quota allocated. By default, every space requires 2 IP addresses and 2 GB of memory to
be allocated to that space to work with the container service.</div><div class="section tsResolve"><div class="p">To distribute the memory and IP quota across multiple spaces:<ol><li><span class="ph">From your account details, in
manage organizations, select an organization.</span></li>
<li><span class="ph">In the quota section, view
the details for containers.</span></li>
<li>Click <span class="ph uicontrol">Edit</span> and allocate the memory and IP addresses across the spaces in
the organization.</li>
<li>Create a container in any space in the organization.</li>
</ol>
</div>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e923" lang="en-us" id="ts_trial_upgrade"><h3 class="topictitle3" id="d81670e923">Quota cannot be edited though you have a pay account</h3>
<div class="body tsBody"><div class="section tsSymptoms">You are not able to modify your container quota.</div><div class="section tsCauses">There might be inconsistencies between the organization's manager and the billing
information.</div><div class="section tsResolve"><div class="p">The logged in user must be the manager of the organization. To verify the owner, complete these steps.<ol><li><span class="ph">From your account details, in
manage organizations, select an organization.</span></li>
<li>In the user list, verify that the manager of the organization is the same as the ID that you are
logging in with.</li>
<li>Log out of the account and log back in.</li>
</ol>
If you complete these steps and you still cannot edit container quota, contact IBM Support.</div>
</div></div>
</article></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e988" lang="en-us" id="ts_ov_images"><h2 class="topictitle2" id="d81670e988">Troubleshooting images</h2>
<div class="body conbody"><p class="shortdesc"></p>
<p></p>
</div>
<aside role="complementary" aria-labelledby="d81670e988"></aside><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1049" lang="en-us" id="ts_firewall_images"><h3 class="topictitle3" id="d81670e1049">Internal proxy blocks requests to push images</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">To create a container within a specific <span class="keyword">Cloud</span> region, you must push the image to
create the container to a registry within that region. When you try to push an image to that
registry, the connection fails with one of the following error messages.
<pre class="pre screen"><code>Connection refused</code></pre>
<pre class="pre screen"><code>Connection timed out</code></pre>
</div><div class="section tsCauses">You might have firewall settings that are preventing communication from your computer to
the <span class="keyword">Cloud</span> registry. Often, internal
proxy settings are configured to allow only permissible communication for security reasons. </div><div class="section tsResolve">In your firewall, open communication from your computer to the registry for the
region-specific IP addresses: <ul id="ts_firewall_images__ul_on4_rfj_wz"><li>US South:
<pre class="codeblock"><code>169.55.39.122
169.55.39.124
169.55.39.126
169.55.211.14
169.46.9.2
169.46.9.3
169.46.9.20
169.46.9.21</code></pre>
</li>
<li>United
Kingdom:<pre class="codeblock"><code>159.8.188.168
159.8.188.169
159.8.188.174
159.8.188.179
169.50.153.69
169.50.153.76
169.50.153.81
169.50.153.85</code></pre>
</li>
</ul>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1114" lang="en-us" id="ts_liberty_image"><h3 class="topictitle3" id="d81670e1114">Problems with the IBM
WebSphere Application Server Liberty images</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">You are having problems with using one of the IBM
WebSphere® Application Server Liberty images in <span class="keyword">IBM
Cloud Container Service</span>.</div><div class="section tsCauses"><p>The image might not be set up correctly.</p>
</div><div class="section tsResolve"><p>If you have issues that relate specifically to this <span class="ph filepath">Dockerfile</span> image, use
the <a href="https://github.com/WASdev/ci.docker/issues" rel="external" target="_blank" title="(Opens in a new tab or window)">GitHub
issue tracker</a>. For more general issues that relate to IBM
WebSphere Application Server Liberty, you can get help
by going to the <a href="https://developer.ibm.com/wasdev/help/" rel="external" target="_blank" title="(Opens in a new tab or window)">WASdev community</a>.</p>
<p>For more information about the IBM
WebSphere Application Server Liberty images in <span class="keyword">IBM
Cloud Container Service</span>, see <a href="../services/RegistryImages/ibmliberty/index.html">IBM
WebSphere Application Server Liberty images</a>.</p>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1243" lang="en-us" id="ts_image_baduserid"><h3 class="topictitle3" id="d81670e1243">Image cannot be pulled from your private <span class="keyword">Cloud</span> images registry</h3>
<div class="body tsBody"><div class="section tsSymptoms">You created a container from a container image in your private <span class="keyword">Cloud</span> registry, but the container gets stuck
in a <span class="ph uicontrol">BUILDING</span> state. When you run the <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> inspect</samp> command, the following error
message is shown in the <span class="ph uicontrol">Env</span> section of your CLI response.
<pre class="codeblock"><code>"Env": [
"metadata_exit_code=-1",
"logging_password=",
"space_id=0e65c436-9413-46e2-8b42-ff06aa74ac21",
"logstash_target=logmet.opvis.bluemix.net:9091",
"metadata_err_msg=Cannot pull image from registry: Pull failed with error message failed
to register layer: ApplyLayer exit status 1 stdout: stderr: Container ID 1234567890
cannot be mapped to a host ID",
"metrics_target=logmet.opvis.bluemix.net:9095",
"metadata_running=1"</code></pre>
</div><div class="section tsCauses">In the <span class="keyword">IBM
Cloud Container Service</span>, the <a href="https://docs.docker.com/engine/security/security/" rel="external" target="_blank" title="(Opens in a new tab or window)">Docker user
namespace feature</a> is enabled in the Docker Engine. The user namespace feature was first
introduced with Docker version 1.10 and provides the possibility to map a root user in the container
to a non-root user outside the container. In this way, the permissions of a user that can perform
root actions inside the container, can be restricted outside the container, so files on the host and
in other containers cannot be accessed, altered, or removed. <div class="steps note"><span class="notetitle">Note:</span> The Docker user namespace is
different to the namespace that you set up in <span class="keyword">Cloud</span> to identify your <a href="container_planning.html#container_planning_namespace" title="To store and manage private images for IBM Cloud Container Service, every organization is required to set up its own private Docker images registry in Cloud. This name of the private registry is called a namespace and must be unique within Cloud.">private images registry</a>.</div>
<p>When you create a container from a container image with the <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> run</samp> command, the image is loaded from
your private <span class="keyword">Cloud</span> images registry
into the IBM Cloud Docker Engine. The Docker Engine then starts to create the respective container
layers. Depending on what is specified in the container image, app files from your local machine
might be added to the container. All these files are created by a user ID (UID) and group ID (GID)
on your local machine. The UID and GID indicate the user's permission on the local machine. When you
want to add these files to the container, the respective UID and GID is mapped to a UID and GID
inside the container. The mapped UID and GID might have different permissions inside the container
than on the local machine. For example, the local machine's non-root UID and GID is mapped to a root
user inside the container. In this way, the user can act like a root user inside the container and
access, alter, and remove container files. As the container is running in an isolated environment,
the UID and GID cannot access any files on the host as it is assigned non-root permissions outside
the container. </p>
<p>To map a local machine's UID and GID to one inside a container, the user
namespace feature requires that all files are created with a UID and GID between 0 and 65536.</p>
<p>The error that is shown occurs when a file is created from a UID or GID outside this range
and when this file is included in one of the container image layers. When you create a container
from the image, the Docker Engine builds each layer of the image. When a file with a UID or GID
outside the range is found, the Docker Engine fails because this UID or GID could not be mapped to a
valid one inside the container. The Docker Engine also fails when a file is used in one image layer
and removed in the next image layer. </p>
</div><div class="section tsResolve"><dl><dt class="dlterm">Enable the user namespace on your local Docker Engine</dt>
<dd>When you build your container images locally, be sure to enable the user namespace feature in
your local Docker Engine. In this way, all UID, and GID on your local machine that are out of the
valid user namespace range are automatically mapped to a valid ID within the range. Test your
container image locally before you push it to your private <span class="keyword">Cloud</span> images registry and use it with <span class="keyword">IBM
Cloud Container Service</span>. For more information about how to
enable the user namespace feature, see <a href="https://docs.docker.com/engine/security/userns-remap/#enable-userns-remap-on-the-daemon" rel="external" target="_blank" title="(Opens in a new tab or window)">Starting the daemon with user namespaces enabled</a>. </dd>
<dt class="dlterm">Comment out the affected image layer and change the ownership of the affected files</dt>
<dd>If you cannot find the file that causes the issue, try to comment out the image layers in your
Dockerfile and uncomment one layer after another. Build your image and create a container to find
the affected image layer. <p>Run <samp class="ph codeph">chown [-R] root:root
<var class="keyword varname"><file_or_folder></var></samp> to change the ownership of the affected file that is
stored on your local machine or add this command to the affected image layer in the Dockerfile.</p>
<p>Example
Dockerfile</p>
<pre class="codeblock"><code>RUN curl -fsSL "https://www.kernel.org/pub/software/scm/git/git-${GIT_VERSION}.tar.gz" \
| tar -xzC /tmp \
&& cd /tmp/git-${GIT_VERSION} \
&& ./configure \
&& make all \
&& make install \
&& chown -R root:root /tmp</code></pre>
</dd>
</dl>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1418" lang="en-us" id="ts_image_export"><h3 class="topictitle3" id="d81670e1418">Image cannot be exported from your private <span class="keyword">Cloud</span> images registry</h3>
<div class="body tsBody"><div class="section tsSymptoms">You are logged in to the <span class="keyword">IBM
Cloud Container Service</span> and want to pull an image from your
private <span class="keyword">Cloud</span> registry to your local
machine by using the <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> pull</samp>
command. The command fails stating that <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> pull</samp> is not supported. </div><div class="section tsCauses">The <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> pull</samp>
command is not supported to pull an image from your private <span class="keyword">Cloud</span> registry to your local machine. </div><div class="section tsResolve">To export an image from your private <span class="keyword">Cloud</span> registry, follow these steps. <ol><li><a href="container_cli_cfic_install.html#container_cli_login" title="After you install the CLI, log in to use it.">Log in to the <span class="keyword">IBM
Cloud Container Service</span></a>. If you are already logged in,
make sure your environment variables are not set to use native Docker commands against the <span class="keyword">IBM
Cloud Container Service</span>. </li>
<li>List your images.
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> images</code></pre>
</li>
<li>Pull an image from your private <span class="keyword">Cloud</span> registry to your local machine. Use the
name or id of the image that was returned in the previous step.
<pre class="codeblock"><code>docker pull <var class="keyword varname"><image_name_or_id></var></code></pre>
</li>
<li>Verify that the image was pulled to your local machine. <pre class="codeblock"><code>docker images</code></pre>
</li>
<li>Save your image into a <span class="ph filepath">.tar</span> file.
<pre class="codeblock"><code>docker export <var class="keyword varname"><image_name_or_id></var> > <var class="keyword varname"><exported_image_name></var>.tar</code></pre>
</li>
</ol>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1575" lang="en-us" id="ts_namespace_change"><h3 class="topictitle3" id="d81670e1575">The namespace for an organization cannot be changed</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">You cannot find a method for changing the namespace.</div><div class="section tsCauses">The namespace for an organization cannot be changed after it is created.</div><div class="section tsResolve">You can create another organization and set the namespace for that new organization to
the desired name.</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1632" lang="en-us" data-hd-audience="" id="ts_image_push_login"><h3 class="topictitle3" id="d81670e1632">Prompted for login during <samp class="ph codeph">docker push</samp></h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you try to run <samp class="ph codeph">docker push</samp> to add an image to your private
<span class="keyword">Cloud</span> images registry, you are
prompted for login credentials. When you enter Docker or <span class="keyword">Cloud</span> credentials, the authentication
fails.</div><div class="section tsCauses">The authentication might fail due to
the following reasons. <ul><li>The bearer token, which contains Cloud
Foundry authentication information, is expired.</li>
<li>The <span class="keyword">Cloud</span> authorization
service
is not available.</li>
</ul>
</div><div class="section tsResolve"><div class="p"><ul><li>If you are using the <span class="keyword">IBM
Cloud Container Service</span> plug-in:<ol><li>Log in to the Cloud Foundry CLI by running the command to refresh the bearer token on your
computer.<pre class="codeblock"><code>bx login [-a api.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>] [-sso]</code></pre>
<span class="ph">The single-sign-on parameter <span class="keyword option">--sso</span> is required when you log in
with a federated ID. If this option is used, open <a href="https://login.ng.bluemix.net/UAALoginServerWAR/passcode" rel="external" target="_blank" title="(Opens in a new tab or window)">https://login.ng.bluemix.net/UAALoginServerWAR/passcode</a> to obtain a one-time passcode. If
you do not include the <span class="keyword option">--sso</span> option, complete these substeps.</span><ol type="a"><li>For <span class="ph uicontrol">Email</span>, enter the IBMId that you
use to log in to <span class="keyword">Cloud</span>.</li>
<li>For <span class="ph uicontrol">Password</span>, enter the password for the IBMId that you use to log in to <span class="keyword">Cloud</span>. By entering your user name and
password, your organization and space are
retrieved.</li>
<li>Enter the number that represents one of your <span class="keyword">Cloud</span> organizations. </li>
<li>Enter the number that represents one of your existing <span class="keyword">Cloud</span> spaces.</li>
</ol>
</li>
<li>Initialize the <span class="keyword">IBM
Cloud Container Service</span> CLI to
re-create your bearer
token.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> init</code></pre>
</li>
</ol>
</li>
</ul>
</div>
<p>If the prompt still occurs after you logged in again, the <span class="keyword">Cloud</span> authorization
service is not available.
Try again later.</p>
</div></div>
</article></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e1804" lang="en-us" id="ts_ov_containers"><h2 class="topictitle2" id="d81670e1804">Troubleshooting containers</h2>
<div class="body conbody"><p class="shortdesc"></p>
<p></p>
</div>
<aside role="complementary" aria-labelledby="d81670e1804"></aside><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1867" lang="en-us" id="ts_group_502"><h3 class="topictitle3" id="d81670e1867"><samp class="ph codeph">502 Bad Gateway</samp> error when accessing an app</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms"><div class="p">You deployed a container group successfully, but when you open the URL for the app, the following
error is
displayed.<pre class="codeblock"><code>502 Bad Gateway: Registered endpoint failed to handle the request.</code></pre>
</div>
</div><div class="section tsCauses"><div class="p"><ol><li>The routing might still be setting up.</li>
<li>The container might not be responding.</li>
<li>There might be an error with the routing though the container group is running
successfully.</li>
</ol>
</div>
</div><div class="section tsResolve"><ol><li>Wait a few more minutes to allow enough time for the routing to complete. There might be a
5-minute wait after running the container process before the container group can connect to the
specified route. </li>
<li>If you have waited 5 minutes and suspect that an error occurred, see if the container is
responding by binding an IP address to the running container and then run a curl command to that IP
address and port.<ol type="a"><li>Run the following command to get the ID of one of the container instances in the group. Note the
container ID and the
port.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> ps</code></pre>
</li>
<li>Request an IP address by running the following
command.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> ip-request</code></pre>
</li>
<li>Bind the IP address to the container instance by running the following
command.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> ip-bind <var class="keyword varname">IP_address</var> <var class="keyword varname">container_ID</var></code></pre>
</li>
<li>In a browser, enter the URL to the application with the IP address and the port. Example:
<span class="ph filepath">http://<var class="keyword varname">IP_address</var>:<var class="keyword varname">port</var></span><p>If you cannot
access the application, then there is a problem with the container group and the group must be
re-created. If you can access the application, the container is running successfully, but there is
an error with the routing. Contact <span class="keyword">IBM
Cloud</span>
Support.</p>
</li>
</ol>
</li>
</ol>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e1984" lang="en-us" id="ts_firewall"><h3 class="topictitle3" id="d81670e1984">Internal proxy blocks requests to the service</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you try to access <span class="keyword">IBM
Cloud Container Service</span>, the connection fails with one
of the following error messages.
<pre class="pre screen"><code>Connection refused</code></pre>
<pre class="pre screen"><code>Connection timed out</code></pre>
</div><div class="section tsCauses">You might have firewall settings that are preventing communication from your computer to
<span class="keyword">IBM
Cloud Container Service</span>. Often, internal
proxy settings are configured to allow only permissible communication for security reasons. </div><div class="section tsResolve">In your firewall, open communication from your computer to <span class="keyword">IBM
Cloud Container Service</span> for the region-specific IP
addresses: <ul id="ts_firewall__ul_fn5_p4l_wz"><li>US South:
<pre class="codeblock"><code>169.46.9.7
169.46.9.19
169.46.9.24
169.46.9.28
169.55.211.15
169.55.211.17
169.55.211.21
169.55.211.28</code></pre>
</li>
<li>London:<pre class="codeblock"><code>159.8.188.181
159.8.188.182
159.8.188.184
159.8.188.188
169.50.153.66
169.50.153.77
169.50.153.91
169.50.153.94</code></pre>
</li>
</ul>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e2065" lang="en-us" id="ts_group_auto_recreate_instance"><h3 class="topictitle3" id="d81670e2065">Container group instances are repeatedly re-created during startup by the <span class="keyword">IBM
Cloud Container Service</span> auto-recovery mode</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">You created a container group and enabled the auto-recovery. During the startup of the
container group, unresponsive group instances are removed and re-created repeatedly.</div><div class="section tsCauses"><p>The app that is running in your container group takes too long to start listening on the exposed
port. If you enabled auto-recovery for your container group, <span class="keyword">IBM
Cloud Container Service</span> checks the health of each instance with
an HTTP request on the exposed TCP port after the group is created. If an instance is not listening
on the exposed port within 3 minutes, the instance is marked unresponsive and is removed,
re-created, and restarted. This process repeats until the maximum number of re-created instances is
reached. Then, auto-recovery is disabled. </p>
</div><div class="section tsResolve"><p>To run an app in a container group with auto-recovery enabled, make sure that the app starts and
listens on the exposed port in less than 2 minutes. Depending on the type of app, different methods
exist to ensure a quick startup of your app.</p>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e2148" lang="en-us" id="ts_bridge_app"><h3 class="topictitle3" id="d81670e2148"><span class="keyword">Cloud</span> service does not bind
directly to a container</h3>
<div class="body tsBody"><div class="section tsSymptoms">Your <span class="keyword">Cloud</span> service does
not bind directly to a container by using service binding.</div><div class="section tsCauses">Some <span class="keyword">Cloud</span> services do not
support direct service binding to a container and lead to a failure during container creation. For
these services, service binding does not pass along the environment variable
<samp class="ph codeph">VCAP_SERVICES</samp> to the container. To work around this problem, you can use the CLI to
create a Cloud Foundry app that acts as a bridge for sending <samp class="ph codeph">VCAP_SERVICES</samp> from the
service to a container. </div><div class="section tsResolve"><ol><li>Select a space in which to deploy a container, such as <samp class="ph codeph">dev</samp>.</li>
<li id="ts_bridge_app__create_cfapp">Identify an existing app or create a Cloud Foundry app to bind the <span class="keyword">Cloud</span> service to. This app is used for the
passing of the <samp class="ph codeph">VCAP_SERVICES</samp> information, so it is referred to as a bridge app.
<div class="p">To create a Cloud Foundry app:<div class="note tip"><span class="tiptitle">Tip:</span> The app that you create by completing these steps
is a placeholder for storing the <samp class="ph codeph">VCAP_SERVICES</samp> services. To incur fewer costs than
a running app, this app does not start or run. The app is not required to run to provide the
<samp class="ph codeph">VCAP_SERVICES</samp> to the container.</div>
</div>
<ol type="a"><li>Create a directory and a text file in your working app directory. Example:
<span class="ph filepath"><var class="keyword varname"><directory></var>/<var class="keyword varname"><empty_file.txt></var></span>. This
file serves as the source file for the app, though it does not need to contain any content.<ul><li>Windows example:
<span class="ph filepath">C:\Users\<var class="keyword varname">User</var>\<var class="keyword varname">containerbridge</var>\<var class="keyword varname">empty_file.txt</var></span></li>
<li>Linux example:
<span class="ph filepath">/<var class="keyword varname">user_home_directory</var>/<var class="keyword varname">containerbridge</var>/<var class="keyword varname">empty_file.txt</var></span></li>
<li>OS X example:
<span class="ph filepath">/Users/<var class="keyword varname">user_home_directory</var>/<var class="keyword varname">containerbridge</var>/<var class="keyword varname">empty_file.txt</var></span></li>
</ul>
</li>
<li>Run this command to create a zero-runtime app at no costs. The app can be named anything though
<var class="keyword varname">containerbridge</var> is a common name since the app acts a bridge from the container
to a
service.<p></p>
<pre class="codeblock" id="ts_bridge_app__containerbridgecode"><code>cf push <var class="keyword varname"><appname></var> -p <var class="keyword varname"><directory></var> -i 1 -d <span class="keyword" data-hd-keyref="APPDomain">AppDomainName</span> -k 1M -m 64M --no-hostname --no-manifest --no-route --no-start</code></pre>
Example:
<pre class="codeblock"><code>cf push <var class="keyword varname">containerbridge</var> -p <var class="keyword varname">containerbridge</var> -i 1 -d <span class="keyword" data-hd-keyref="APPDomain">AppDomainName</span> -k 1M -m 64M --no-hostname --no-manifest --no-route --no-start</code></pre>
</li>
</ol>
</li>
<li>If the <span class="keyword">Cloud</span> service you want to
bind isn't already added, add the service to the <span class="keyword">Cloud</span> space.<ol type="a"><li>To get a list of the available <span class="keyword">Cloud</span> services, run the <samp class="ph codeph">cf
marketplace</samp> command. The resulting list includes the name of the <span class="keyword">Cloud</span> service, its plan type, and a
description of what the service is.</li>
<li>Run the <samp class="ph codeph">create-service</samp> command and include the service name and plan from the
<samp class="ph codeph">cf marketplace</samp> list. Also, include your own instance name for the
service.<pre class="codeblock"><code>cf create-service <var class="keyword varname"><service></var> <var class="keyword varname"><plan></var> <var class="keyword varname"><service_instance></var></code></pre>
Example<pre class="codeblock"><code>cf create-service <var class="keyword varname">cleardb</var> <var class="keyword varname">spark</var> <var class="keyword varname">my-clear-db</var></code></pre>
</li>
</ol>
</li>
<li>Bind the <span class="keyword">Cloud</span> service to the app
by running the bind-service command. Repeat to bind other <span class="keyword">Cloud</span> services to the bridge
app.<pre class="codeblock"><code>cf bind-service <var class="keyword varname"><appname></var> <var class="keyword varname"><service_instance></var></code></pre>
Example<pre class="codeblock"><code>cf bind-service <var class="keyword varname">containerbridge</var> <var class="keyword varname">my-clear-db</var></code></pre>
</li>
<li>If you are prompted to restage the app, run the following
command.<pre class="codeblock"><code>cf restage <var class="keyword varname">containerbridge</var></code></pre>
</li>
<li>Review the information for the <span class="keyword">Cloud</span> service that is passed through the
<var class="keyword varname">containerbridge</var> app by running the following
command.<pre class="codeblock"><code>cf env <var class="keyword varname"><appname></var> </code></pre>
Example<pre class="codeblock"><code>cf env <var class="keyword varname">containerbridge</var> </code></pre>
<p>The
results include the injected <span class="keyword">Cloud</span>
service parameters by using the <samp class="ph codeph">VCAP_SERVICES</samp> environment variable. </p>
<div class="p">For
example:
<pre class="codeblock"><code>{
"VCAP_SERVICES": {
"<service_name>": [
{
"credentials": {
"hostname": "<hostname_id>",
"jdbcUrl": "<jdbc_URL>",
"name": "<encoded_name>",
"password": "<encoded_password>",
"port": "<port_number>",
"uri": "<service_URI>",
"username": "<encoded_username>"
},
"label": "<service_name>",
"name": "<service_instance>",
"plan": "<plan_name>",
"tags": [
"<tag1>",
"<tag2>",
"<tag3>"
]
} </code></pre>
</div>
</li>
</ol>
<div class="p">After you create the binding between the <span class="keyword">Cloud</span> services and the
<var class="keyword varname">containerbridge</var> app, create a container and bind the
<var class="keyword varname">containerbridge</var> app to it. For more information, see the
<samp class="ph codeph">CCS_BIND_APP</samp> flag in the environment variables in <a href="container_single_ui.html#container_single_cli" title="Create a container by using the run command. A container must include an image in it. If you do not have one yet, you can use one of the default IBM certified images that are available by default in your organization's private images registry.">Creating a single container</a> or <a href="container_ha.html#container_group_cli" title="Create and deploy a scalable group container from the IBM Cloud Container Service CLI. A container group includes two or more containers that run the same image. Use container groups for running long-term services with workloads that require scalability and reliability or for testing at the required scale.">Creating a container
group</a>.<div class="note tip" id="ts_bridge_app__delete_bound_app"><span class="tiptitle">Tip:</span> After the bridge app is bound to a container, do
not delete the bridge app. If you delete a bound bridge app, the container hangs. If you no longer
require the bridge app, delete the container and any other containers that are using the bridge app
before you delete the bridge app.</div>
</div>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e2490" lang="en-us" id="su_error"><h3 class="topictitle3" id="d81670e2490"><samp class="ph codeph">Cannot create child process</samp> with <samp class="ph codeph">su</samp> command</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms"><p>When using the substitute user (<samp class="ph codeph">su</samp>) Linux command to edit the users
for a container, you receive the message <samp class="ph codeph">su: cannot create child process: Resource
temporarily unavailable</samp>, even though the <samp class="ph codeph">su</samp> command worked in the local
Docker environment. You might experience this issue when you log into a container or when the
<samp class="ph codeph">su</samp> command is run as part of a script within the container. </p>
<p>Example:</p>
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> exec -it <var class="keyword varname">my_container</var> bash</code></pre>
<pre class="codeblock"><code>adduser <var class="keyword varname">user</var></code></pre>
<pre class="codeblock"><code>su - <var class="keyword varname">user</var></code></pre>
Result:<pre class="codeblock"><code>su: cannot create child process: Resource temporarily unavailable</code></pre>
</div><div class="section tsCauses"><p>This issue often occurs in containers that are built with Centos 7 images, though it might occur
with other image types as well. </p>
</div><div class="section tsResolve">You can resolve the issue by editing the Dockerfile and re-creating the image or by
logging in to the running container.<div class="p">To resolve the issue by editing the Dockerfile:<ol><li>In the local directory you are creating your image from, create an <span class="ph filepath">etc</span>
folder, if one does not exist already.<pre class="codeblock"><code>mkdir etc</code></pre>
</li>
<li>Navigate to the <samp class="ph codeph">etc</samp> directory.<pre class="codeblock"><code>cd etc</code></pre>
</li>
<li>In the <span class="ph filepath">etc</span> folder, create a <span class="ph filepath">pam.d</span>
directory.<pre class="codeblock"><code>mkdir pam.d</code></pre>
</li>
<li>Navigate to the <samp class="ph codeph">pam.d</samp> directory.<pre class="codeblock"><code>cd pam.d</code></pre>
</li>
<li>In the <span class="ph filepath">pam.d</span> directory, create a file named <span class="ph filepath">su</span> and
open it with a text editor.</li>
<li>In the <span class="ph filepath">su</span> file, copy in the following text where all of the
<samp class="ph codeph">session</samp> variables are set to <samp class="ph codeph">optional</samp> and save the changes to the
file.
<pre class="codeblock"><code>#%PAM-1.0
auth sufficient pam_rootok.so
# Uncomment the following line to implicitly trust users in the "wheel" group.
#auth sufficient pam_wheel.so trust use_uid
# Uncomment the following line to require a user to be in the "wheel" group.
#auth required pam_wheel.so use_uid
auth substack system-auth
auth include postlogin
account sufficient pam_succeed_if.so uid = 0 use_uid quiet
account include system-auth
password include system-auth
session optional system-auth
session optional postlogin
session optional pam_xauth.so</code></pre>
</li>
<li>In the Dockerfile for the image, include the following
information.<pre class="codeblock"><code>RUN sed -i 's/session.*include/session optional/' /etc/pam.d/su</code></pre>
</li>
<li>Re-create the image. For more information, see <a href="container_images_building.html" title="Create a Dockerfile to build an image. A Dockerfile is a file that includes specifications for building a Docker image, which is the basis for a container. After you create the Dockerfile, you can build the image locally or directly into the private Cloud registry by using the command-line interface. Then, the image is available through the Catalog view in Cloud.">Building an image</a>.</li>
<li>Re-create the single container or container group. For more information, see <a href="container_single_ui.html#container_single_cli" title="Create a container by using the run command. A container must include an image in it. If you do not have one yet, you can use one of the default IBM certified images that are available by default in your organization's private images registry.">Creating a single container</a> or <a href="container_ha.html#container_group_cli" title="Create and deploy a scalable group container from the IBM Cloud Container Service CLI. A container group includes two or more containers that run the same image. Use container groups for running long-term services with workloads that require scalability and reliability or for testing at the required scale.">Creating a container
group</a>.</li>
<li>To verify, run the following
command.<pre class="codeblock"><code><span class="ph">bx ic</span> logs <var class="keyword varname">container_name</var></code></pre>
Result:<pre class="codeblock"><code>%Changing password for user <var class="keyword varname">user_name</var>.
passwd: all authentication tokens updated successfully.</code></pre>
</li>
</ol>
</div>
<div class="p">To resolve the issue by logging in to the running container:<ol><li>Create the container.</li>
<li>Log into the
container.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> exec -it <var class="keyword varname">my_container</var> bash</code></pre>
</li>
<li>Navigate to the <samp class="ph codeph">etc</samp> directory.<pre class="codeblock"><code>cd etc</code></pre>
</li>
<li>Navigate to the <samp class="ph codeph">pam.d</samp> directory.<pre class="codeblock"><code>cd pam.d</code></pre>
</li>
<li>Open the visual editor by using the Linux <samp class="ph codeph">vi</samp>
command.<pre class="codeblock"><code>vi su</code></pre>
</li>
<li>In the <span class="ph filepath">su</span> file, change all of the session variables to
<samp class="ph codeph">optional</samp> and save the changes to the file. For more information about using the
<samp class="ph codeph">vi</samp> editor, see <a href="https://www.washington.edu/computing/unix/vi.html" rel="external" target="_blank" title="(Opens in a new tab or window)">How to Use the vi
Editor</a>.<pre class="codeblock"><code>#%PAM-1.0
auth sufficient pam_rootok.so
# Uncomment the following line to implicitly trust users in the "wheel" group.
#auth sufficient pam_wheel.so trust use_uid
# Uncomment the following line to require a user to be in the "wheel" group.
#auth required pam_wheel.so use_uid
auth substack system-auth
auth include postlogin
account sufficient pam_succeed_if.so uid = 0 use_uid quiet
account include system-auth
password include system-auth
session optional system-auth
session optional postlogin
session optional pam_xauth.so</code></pre>
</li>
<li>Log in as the user.<pre class="codeblock"><code>su - <var class="keyword varname">user</var></code></pre>
</li>
<li>Run the <samp class="ph codeph">id</samp> Linux command to see the configuration information for the
user.<pre class="codeblock"><code>id</code></pre>
Result:<pre class="codeblock"><code>uid=1000(<var class="keyword varname">user</var>) gid=1000(<var class="keyword varname">user</var>) groups=1000(<var class="keyword varname">user</var>)</code></pre>
</li>
</ol>
</div>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e2786" lang="en-us" id="ts_container_shutdown"><h3 class="topictitle3" id="d81670e2786">Container stops and shuts down after creation</h3>
<div class="body tsBody"><p class="shortdesc">You created a container, but it shut down immediately.</p>
<div class="section tsSymptoms">You ran a container locally and it ran successfully, but when you ran the container in
<span class="keyword">Cloud</span>, you see one of the following situations.<ul><li>From the <span class="keyword">Cloud</span> dashboard, the
container state changes from <samp class="ph codeph">Running</samp> to <samp class="ph codeph">Stopped</samp>.</li>
<li>From the CLI, when you run <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span>
inspect container</samp>, you see that the state status is <samp class="ph codeph">Shutdown</samp>. </li>
</ul>
You also tried running <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span>
logs</samp>, but did not find any issues with the container.</div><div class="section tsCauses"><p>When you run a container in <span class="keyword">Cloud</span>,
the container is always run detached mode, similar to running <samp class="ph codeph">docker run -d</samp> when a
container is run locally. As a result, even containers that start locally do not run continuously in
<span class="keyword">Cloud</span>.</p>
</div><div class="section tsResolve"><div class="p">You must make the container a long-running application by editing the Dockerfile. For example, if
you added a command to output the phrase "hello world" every ten seconds, that continuous action
makes your container a long-running
application.<pre class="codeblock"><code>ENTRYPOINT ["/bin/sh"]
CMD ["-c","while true; do echo hello world; sleep 10;done"]</code></pre>
</div>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e2895" lang="en-us" id="ts_incomplete_deletion"><h3 class="topictitle3" id="d81670e2895">Deleted container continues to show in the <span class="keyword">Cloud</span> GUI</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">From the <span class="keyword">Cloud</span> GUI, you
deleted a container, but the container continues to be displayed in the <span class="keyword">Cloud</span> GUI and uses quota.</div><div class="section tsCauses"><p>From the <span class="keyword">Cloud</span> GUI, the deletion
could not be completed.</p>
</div><div class="section tsResolve"><p>Delete the container from the command line.</p>
<div class="p">For single containers, follow these steps.<ol><li>Delete the container by running one of the following command line options.<ul><li><span class="ph"><a href="container_cli_cfic_install.html#container_cli_login__option1_li" rel="external" target="_blank" title="During bx ic login, you selected Option 1 to use bx ic commands to manage containers. Docker commands are used only for local container development. (Opens in a new tab or window)"><img src="images/option1.png" alt="BX IC plugin configuration option 1"></a>
<span class="keyword">IBM
Cloud Container Service</span> plug-in with <span class="ph"><samp class="ph codeph">bx ic</samp></span> commands </span><pre class="codeblock"><code><span class="ph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> rm [-f] CONTAINER [CONTAINER] </span></code></pre>
</li>
<li><div class="p"><span class="ph"><a href="container_cli_cfic_install.html#container_cli_login__option2_li" rel="external" target="_blank" title="During bx ic login, you selected Option 2 by configuring environment variables to use docker commands whenever possible to manage containers. bx ic commands are used only when no supported Docker alternative exists. (Opens in a new tab or window)"><img src="images/option2.png" alt="BX IC plugin configuration option 2"></a>
<span class="keyword">IBM
Cloud Container Service</span> plug-in with
<samp class="ph codeph">docker</samp> commands</span>
<pre class="codeblock"><code>docker rm [-f] CONTAINER</code></pre>
</div>
</li>
</ul>
</li>
<li>Verify that the container was removed by running one of the following commands to list all of
the containers in your space:<ul><li><span class="ph"><a href="container_cli_cfic_install.html#container_cli_login__option1_li" rel="external" target="_blank" title="During bx ic login, you selected Option 1 to use bx ic commands to manage containers. Docker commands are used only for local container development. (Opens in a new tab or window)"><img src="images/option1.png" alt="BX IC plugin configuration option 1"></a>
<span class="keyword">IBM
Cloud Container Service</span> plug-in with <span class="ph"><samp class="ph codeph">bx ic</samp></span> commands </span><pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> ps -a</code></pre>
</li>
<li><span class="ph"><a href="container_cli_cfic_install.html#container_cli_login__option2_li" rel="external" target="_blank" title="During bx ic login, you selected Option 2 by configuring environment variables to use docker commands whenever possible to manage containers. bx ic commands are used only when no supported Docker alternative exists. (Opens in a new tab or window)"><img src="images/option2.png" alt="BX IC plugin configuration option 2"></a>
<span class="keyword">IBM
Cloud Container Service</span> plug-in with
<samp class="ph codeph">docker</samp> commands</span>
<pre class="codeblock"><code>docker ps -a</code></pre>
</li>
</ul>
</li>
</ol>
</div>
<div class="p">For container groups, follow these steps. <ol><li>Delete the container group by running the following command.<ul><li><span class="ph"><a href="container_cli_cfic_install.html#container_cli_login__option1_li" rel="external" target="_blank" title="During bx ic login, you selected Option 1 to use bx ic commands to manage containers. Docker commands are used only for local container development. (Opens in a new tab or window)"><img src="images/option1.png" alt="BX IC plugin configuration option 1"></a>
<a href="container_cli_cfic_install.html#container_cli_login__option2_li" rel="external" target="_blank" title="During bx ic login, you selected Option 2 by configuring environment variables to use docker commands whenever possible to manage containers. bx ic commands are used only when no supported Docker alternative exists. (Opens in a new tab or window)"><img src="images/option2.png" alt="BX IC plugin configuration option 2"></a>
<span class="keyword">IBM
Cloud Container Service</span> plug-in</span><pre class="codeblock"><code><span class="ph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-remove [-f] GROUP [GROUP]</span></code></pre>
</li>
</ul>
</li>
<li>Verify that the container was removed by running the command to list all of the containers in
your space.<ul><li><span class="ph"><a href="container_cli_cfic_install.html#container_cli_login__option1_li" rel="external" target="_blank" title="During bx ic login, you selected Option 1 to use bx ic commands to manage containers. Docker commands are used only for local container development. (Opens in a new tab or window)"><img src="images/option1.png" alt="BX IC plugin configuration option 1"></a>
<a href="container_cli_cfic_install.html#container_cli_login__option2_li" rel="external" target="_blank" title="During bx ic login, you selected Option 2 by configuring environment variables to use docker commands whenever possible to manage containers. bx ic commands are used only when no supported Docker alternative exists. (Opens in a new tab or window)"><img src="images/option2.png" alt="BX IC plugin configuration option 2"></a>
<span class="keyword">IBM
Cloud Container Service</span> plug-in</span><pre class="codeblock"><code><span class="ph"><span class="ph"><samp class="ph codeph">bx ic</samp></span> groups [-q] </span></code></pre>
</li>
</ul>
</li>
</ol>
</div>
<p>If you run the commands and the container continues to display in your <span class="keyword">Cloud</span> GUI, wait to see whether it gets removed
later. </p>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e3231" lang="en-us" id="ts_group_network_failure"><h3 class="topictitle3" id="d81670e3231">Container group is showing a HAS_INACTIVE_INSTANCES status</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you run <samp class="ph codeph"><span class="ph"><samp class="ph codeph">bx ic</samp></span>
group-inspect</samp>, the container group is displaying a HAS_INACTIVE_INSTANCES
status.</div><div class="section tsCauses"><p>If the networking fails to be configured when the group is created or fails the health check for
one or more instances, the instance state is change to <samp class="ph codeph">INACTIVE</samp>. If any instance in
a group is inactive, the group state is changed to HAS_INACTIVE_INSTANCES.</p>
</div><div class="section tsResolve"><p>If the issue occurs when the health check fails for an instance, the issue can resolve without
any action required. If the issue occurs when the group is first created, you might try re-creating
the group. If the issue occurs again after re-creating, networking might not be configurable for the
app when it is deployed as a group. Try deploying the app as a single container instead.</p>
</div></div>
</article></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e3304" lang="en-us" id="ts_ov_compose"><h2 class="topictitle2" id="d81670e3304">Troubleshooting Docker Compose</h2>
<div class="body conbody"><p class="shortdesc"></p>
<p></p>
</div>
<aside role="complementary" aria-labelledby="d81670e3304"></aside><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e3365" lang="en-us" id="ts_compose_insecure_connection"><h3 class="topictitle3" id="d81670e3365">Insecure HTTPS connection warning is displayed when executing Docker Compose commands</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you run Docker Compose commands in the CLI, a warning similar to one of the
following examples is displayed. Other related error messages can be found in the <a href="https://urllib3.readthedocs.org/en/latest/security.html" rel="external" target="_blank" title="(Opens in a new tab or window)">urlib3
development documentation</a>.<div class="p"><pre class="pre msgblock"><code>/.../requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request
has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this
platform. This may cause the server to present an incorrect TLS certificate, which can cause
validation failures. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning. SNIMissingWarning</code></pre>
<pre class="pre msgblock"><code>/.../requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true
SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately
and may cause certain SSL connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning</code></pre>
</div>
</div><div class="section tsCauses">This error message is typical for Linux operating systems and is caused by a missing
verification of HTTPS requests. </div><div class="section tsResolve"><ul><li>Option 1: Install the <samp class="ph codeph">ndg-httpsclient</samp> to avoid SSL related error messages and
warnings.<pre class="codeblock"><code>pip install ndg-httpsclient==0.4.0</code></pre>
</li>
<li>Option 2: If the <samp class="ph codeph">ndg-httpsclient</samp> is already installed on your computer, upgrade
the package to the latest version. <pre class="codeblock"><code>pip install --upgrade ndg-httpsclient</code></pre>
</li>
</ul>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e3446" lang="en-us" id="ts_compose_commands_failed"><h3 class="topictitle3" id="d81670e3446">No module named queue error is displayed when executing Docker Compose commands</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">As soon as you run Docker Compose commands in your CLI, an error message similar to the
following example is displayed. <div class="p"><pre class="pre msgblock"><code>File "/usr/local/bin/docker-compose", line 9, in <module>
load_entry_point('docker-compose==1.6.2', 'console_scripts', 'docker-compose')() File
"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 357,
in load_entry_point return get_distribution(dist).load_entry_point(group, name) File
"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 2394,
in load_entry_point return ep.load() File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py",
line 2108, in load entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/Library/Python/2.7/site-packages/compose/cli/main.py", line 26, in <module> from ..project import NoSuchService
File "/Library/Python/2.7/site-packages/compose/project.py", line 10, in <module> from . import parallel
File "/Library/Python/2.7/site-packages/compose/parallel.py", line 9, in <module> from six.moves.queue import Empty ImportError:
No module named queue</code></pre>
</div>
</div><div class="section tsCauses">The Python path is pointing to the wrong directory. The Docker Compose Python package
cannot be found. </div><div class="section tsResolve">Change the Python path to the correct directory.
<pre class="codeblock"><code>export PYTHONPATH=/Library/Python/2.7/site-packages/:$PYTHONPATH</code></pre>
</div></div>
</article></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e3509" lang="en-us" id="ts_ov_volumes"><h2 class="topictitle2" id="d81670e3509">Troubleshooting data storage</h2>
<div class="body conbody"><p class="shortdesc"></p>
<p></p>
</div>
<aside role="complementary" aria-labelledby="d81670e3509"></aside><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e3572" lang="en-us" id="ts_vol_owner"><h3 class="topictitle3" id="d81670e3572">Permission error changing the owner of a mounted volume directory in a container</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">You get a permission error when you try to change the ownership of a file or directory
for a volume that is mounted in a container.</div><div class="section tsCauses"><span class="ph" id="ts_vol_owner__containers_namespace">For <span class="keyword">IBM
Cloud Container Service</span>, the <a href="https://docs.docker.com/engine/security/security/#kernel-namespaces" rel="external" target="_blank" title="(Opens in a new tab or window)">user namespace feature</a> is enabled for Docker Engine.</span> The effective root
inside the container is mapped to a non-root user on the compute host. The source of the mounted
volume is a network file system (NFS). Since the NFS is not aware of user namespaces, all non-user
checks are performed for the operations on mounted files and directories inside the container. </div><div class="section tsResolve"><p></p>
<ol><li>In the Dockerfile for an image, add the following lines to create a user with an available user
ID (UID). For example, you can name the user <var class="keyword varname">volumeeditor</var>.<pre class="codeblock"><code>RUN groupadd --gid <var class="keyword varname">1010</var> <var class="keyword varname">volumeeditor</var>
RUN useradd --uid <var class="keyword varname">1010</var> --gid <var class="keyword varname">1010</var> -m --shell /bin/bash <var class="keyword varname">volumeeditor</var></code></pre>
</li>
<li>Build and push the image to your private <span class="keyword">Cloud</span> images registry.</li>
<li>Create a container with the image where the volume path is mounted.<div class="note tip"><span class="tiptitle">Tip:</span> <span class="ph">Run <samp class="ph codeph"><span class="ph">bx ic</span> namespace-get</samp> to retrieve
your namespace and replace <var class="keyword varname"><my_namespace></var> with your namespace information.</span></div>
<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> run --volume <var class="keyword varname">volume_name</var>:<var class="keyword varname">/path/to/files</var> --name <var class="keyword varname">fixVolumePermissions</var> registry.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>/<var class="keyword varname"><my_namespace></var>/<var class="keyword varname">image</var></code></pre>
</li>
<li>Log in to the
container.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> exec -it <var class="keyword varname">container_name</var> bash</code></pre>
</li>
<li>Define an environment variable that is named <kbd class="ph userinput">MOUNTPATH</kbd> for the mount
path.<pre class="codeblock"><code>MOUNTPATH="<var class="keyword varname">/path/to/files</var>"</code></pre>
</li>
<li>Add the user to the group
<samp class="ph codeph">root</samp>.<pre class="codeblock"><code>adduser <var class="keyword varname">volumeeditor</var> root</code></pre>
</li>
<li>Set permissions so that all group members have read and write access to the volume mount.
<pre class="codeblock"><code>chmod 775 $MOUNTPATH</code></pre>
</li>
<li>Create a folder in the mountpath and set the read and write permissions. In this example the
folder is named
<var class="keyword varname">data</var>.<pre class="codeblock"><code>su -c "mkdir -p $MOUNTPATH/<var class="keyword varname">data</var>" <var class="keyword varname">volumeeditor</var></code></pre>
<pre class="codeblock"><code>su -c "chmod 700 $MOUNTPATH/<var class="keyword varname">data</var>" <var class="keyword varname">volumeeditor</var></code></pre>
</li>
<li>Verify that the user <var class="keyword varname">volumeeditor</var> can access the files and directories in the
mount path.<pre class="codeblock"><code>sudo -u <var class="keyword varname">volumeeditor</var> ls -al $MOUNTPATH</code></pre>
</li>
<li>Optional: Link a folder in the container to the <var class="keyword varname">data</var> folder in the volume.
This link associates the write permissions of $MOUNTPATH/<var class="keyword varname">data</var> with any folder in
the
container.<pre class="codeblock"><code>ln -sf $MOUNTPATH/<var class="keyword varname">data</var> <var class="keyword varname">/folder/in/container/</var></code></pre>
</li>
<li> If the edits can be made without a permission error, remove the <var class="keyword varname">volumeeditor</var>
group root user.<pre class="codeblock"><code>deluser <var class="keyword varname">volumeeditor</var> root</code></pre>
</li>
</ol>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e3795" lang="en-us" id="ts_vol_group_overwrite"><h3 class="topictitle3" id="d81670e3795">Logs from container group instances are overwritten in a volume</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you created a container group, you mounted a volume and have logs outputted from
each container group instance to the volume. Each container group instance writes to the same files
and the files are overwritten by each of the instances.
<div class="p">Example:<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-create –p 9080 –m <var class="keyword varname">64</var> -v log_vol:/opt/ibm/logs -e LOG_LOCATIONS=/opt/ibm/logs/messages.log,/opt/ibm/logs/debug.log,/opt/ibm/logs/trace.log -–name <var class="keyword varname">my_container_group</var> registry.<var class="keyword varname" data-hd-keyref="DomainName">domain_name</var>/ibmliberty:latest</code></pre>
</div>
</div><div class="section tsCauses">Each container instance is writing a file to the volume as if it were the sole writer.
Each instance has a local cache in memory. The files in the volume are repeatedly replaced by the
cached data from each container instance.</div><div class="section tsResolve">Adjust the name of the log file to include an identifier so that the files are different
for each instance. For example, include the <samp class="ph codeph">$HOSTNAME</samp> variable in the log file
name that is generated by the app and specified when the container group is
created.<div class="p">Example:<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-create –p 9080 –m <var class="keyword varname">64</var> -v log_vol:/opt/ibm/logs -e LOG_LOCATIONS=/opt/ibm/logs/messages.$HOSTNAME.log,/opt/ibm/logs/debug.$HOSTNAME.log,/opt/ibm/logs/trace.$HOSTNAME.log -–name <var class="keyword varname">my_container_group</var> registry.<var class="keyword varname" data-hd-keyref="DomainName">domain_name</var>/ibmliberty:latest</code></pre>
</div>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e3886" lang="en-us" id="ts_vol_performance"><h3 class="topictitle3" id="d81670e3886">Transferring data to or from a volume takes longer than expected</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">Reading to or writing from a volume takes longer than expected.</div><div class="section tsCauses">In <span class="keyword">IBM
Cloud Container Service</span>, volumes do not
have access to the local host. Therefore, all volumes are mounted on a NFS file share. The
performance of the volume is dictated by the size and IOPS (input/output per second) of your file
share, along with the work load of your application. The file share your volume is hosted on may not
be suited for the workload of your volumes and containers. In addition, multiple volumes on one file
share can compete for the file share's resources.<p>To create a new file share, you must have the
organization manager role. For detailed information, see <a href="../iam/users_roles.html">Users and roles</a>.</p>
</div><div class="section tsResolve"><ol><li>Determine the ID of the file share that your volume is hosted
on.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> volume-inspect <var class="keyword varname">my_volume</var></code></pre>
<pre class="codeblock"><code>{
"fsId": "<var class="keyword varname">fa277ff4-8a64-435b-9b75-0f11d59a3ae4</var>",
"hostPath": "<span class="ph filepath">/vol/fa277ff4-8a64-435b-9b75-0f11d59a3ae4/my_volume</span>",
"otherSpaceVisibility": [],
"spaceGuid": "fa277ff4-8a64-435b-9b75-0f11d59a3ae4",
"volName": "<var class="keyword varname">my_volume</var>"
}</code></pre>
</li>
<li>View your file share to determine its size and
IOPS.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> volume-fs</code></pre>
<pre class="codeblock"><code>Name Size IOPS/GB Created State
<var class="keyword varname">fa277ff4-8a64-435b-9b75-0f11d59a3ae4</var> 20 4.00 2016-05-20 13:48:10 READY </code></pre>
</li>
<li>View the available sizes in GB for your file
shares.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> volume fs-flavor</code></pre>
<pre class="codeblock"><code>[
20,
40,
80,
100,
250,
500,
1000,
2000,
4000,
8000,
12000
]</code></pre>
</li>
<li>Determine the size and IOPS that you need for your apps' workload. Valid IOPS values are 0.25, 2
or 4 IOPS per GB. The file share size and IOPS per GB multiply together to determine the file
share's total IOPS. This table includes examples of how different file share sizes at 4 IOPS per GB
scale. <div class="tablenoborder"><table summary="" id="ts_vol_performance__table_zw1_2f3_kw" class="defaultstyle"><caption><span class="tablecap">Table 1. Examples of file share sizes and IOPS per GB</span></caption><thead><tr><th id="d81670e4005" class="thleft"> File share sizes</th>
<th id="d81670e4007" class="thleft">IOPS per GB</th>
<th id="d81670e4009" class="thleft">Total IOPS</th>
</tr>
</thead>
<tbody><tr><td headers="d81670e4005 ">20 GB</td>
<td headers="d81670e4007 ">4</td>
<td headers="d81670e4009 ">80</td>
</tr>
<tr><td headers="d81670e4005 ">100 GB</td>
<td headers="d81670e4007 ">4</td>
<td headers="d81670e4009 ">400</td>
</tr>
<tr><td headers="d81670e4005 ">1,000 GB</td>
<td headers="d81670e4007 ">4</td>
<td headers="d81670e4009 ">4,000</td>
</tr>
<tr><td headers="d81670e4005 ">4,000 GB</td>
<td headers="d81670e4007 ">4</td>
<td headers="d81670e4009 ">16,000</td>
</tr>
</tbody>
</table>
</div>
</li>
<li>Create a new file share. The first number is the size in GB. The second numbers is the file
share
IOPS.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> volume-fs-create <var class="keyword varname">my_new_fs</var> <var class="keyword varname">1000</var> <var class="keyword varname">4</var></code></pre>
</li>
<li>Verify the new file share and wait for the status to show as
ready.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> volume-fs</code></pre>
</li>
<li>Create your volume on the new file
share.<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> volume-create <var class="keyword varname">my_new_volume</var> <var class="keyword varname">my_new_fs</var></code></pre>
</li>
<li>Run a container or container group from your image with the new volume mounted.</li>
<li>Back up data from <var class="keyword varname">my_volume</var> and copy the data to
<var class="keyword varname">my_new_volume</var>. See <a href="container_volumes_ov.html#container_volumes_backup" title="Container volumes are hosted on persistent storage but do not have backups. If you require a backup of your volume data, you can use bx ic cp to create a manual backup.">Backing up and restoring data</a> for more information.</li>
</ol>
</div></div>
</article><article class="topic tsTroubleshooting nested2" role="article" aria-labelledby="d81670e4096" lang="en-us" id="ts_vol_mount_option"><h3 class="topictitle3" id="d81670e4096">Parsing error when creating a container with a volume from the CLI</h3>
<div class="body tsBody"><p class="shortdesc"></p>
<div class="section tsSymptoms">When you create a single container or container group from the CLI, you receive a failed
message.<div class="p">Example<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-create --name my_container -n my_host -d <span class="keyword" data-hd-keyref="APPDomain">AppDomainName</span> -m 2048 --min 1 --max 2 --desired 1 -p 9080 --auto -v my_volume:/my/path registry.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>/ibmliberty</code></pre>
</div>
<div class="p">Result<pre class="codeblock"><code>docker: Error parsing reference: “my_volume:/my/path” is not a valid repository/tag.</code></pre>
</div>
</div><div class="section tsCauses">The CLI for Cloud Foundry version 6.17 and later uses the <span class="keyword option">-v</span> parameter
for another purpose.</div><div class="section tsResolve">In your command, change the <span class="keyword option">-v</span> option to the <span class="keyword option">--volume</span>
option.<div class="p">Example<pre class="codeblock"><code><span class="ph"><samp class="ph codeph">bx ic</samp></span> group-create --name my_container -n my_host -d <span class="keyword" data-hd-keyref="APPDomain">AppDomainName</span> -m 2048 --min 1 --max 2 --desired 1 -p 9080 --auto --volume my_volume:/my/path registry.<span class="keyword" data-hd-keyref="DomainName">DomainName</span>/ibmliberty</code></pre>
</div>
</div></div>
</article></article><article class="topic concept nested1" role="article" aria-labelledby="d81670e4191" lang="en-us" id="ts_known_issues"><h2 class="topictitle2" id="d81670e4191">Known issues</h2>