Create test cluster with H100

[tssala23]

Motivation

We want to test that H100s are fully operational in our existing OCP setup

Completion Criteria

Ensure we are able to run workloads with and without OAI

Description

• Create cluster using ESI nodes including new H100 nodes
• configure cluster
• Run openshift tests on cluster

Completion dates

Desired - 20YY-MM-DD
Required - TBD

[joachimweyl]

@tssala23 is the plan to create a whole new cluster for testing or to just use one of our already standing clusters to test?

[tssala23]

@joachimweyl I guess either could work. Currently cluster are being used by people for stuff so I could opt to create a new one. Unless we added one to the test cluster but I am not able to add and remove nodes from that, just Justin from my understanding.

[joachimweyl]

If all the other test clusters are in use then we might as well spin up a new one.

[joachimweyl]

How long are you thinking this cluster will take to spin up?

[tssala23]

Not long, same process as all other clusters.
But thinking about it, will probably just be easier to attach it to one of the cluster built with esi, like Albany or Barcelona. There's also Cairo that Dylan is building. That way we don't have to worry about DNS records

[joachimweyl]

@tssala23 please research which ones are in use for what purposes and if there is one available please use that and update this issue to add the H100s to that cluster. Otherwise please continue with the process of spinning up a new "Dublin" or "Damascus"?

[tssala23]

Albany - Isaiah was using for GPU Metrics and performance stuff
Barcelona - Danni for distributed instruct lab
Cairo - Waiting for DNS but I dont think it has been assigned.
So thinking about it, it is likely we wont need to create a new one, as I dont think Cairo has a purpose yet, but I'm sure I could also share one of the other two, though I do like the name "Damascus".
I will check with Danni and Isaiah about their usage.

[hpdempsey]

This is the spreadsheet where we track the testbed useage.  @dystewart should be keeping it updated so you can track @joachimweyl along with me and others.   ;-)

It is not yet clear to me how the H100s will be allocated to clusters.   I believe Wayne said he wanted some for OpenStack.  In addition to what Taj is describing here, which I believe would be our first attempt to run H100s with our existing production versions of OCP/RHOAI software, we would also want to try them with the newest versions of both software, which is ideally happening in the second production cluster, but if not that, then we will need a test cluster for that.

Jason, Ahmed and Sanjay will also need access to H100s.  They will likely be bare metal (at least at first) and not clusters.  We will want at least two of the first H100s to go to them for this testing, and they will need to use the new switches for these tests with ROCE. We will also need to test the new ACM and observability software on some cluster with updated software and H100s as well. This testing might be able to co-exist with one of the other test clusters if planned carefully (that is what the existing spreadsheet assumed.)

If there is a second production cluster approved (soon), then we will need to forecast H100s for that too.
Since we don't know yet how many H100s will survive burn-in (do we?), I don't know how many we get to allocate where and how Hakan intends to network them for the various opportunities, given the limited number of switches.
Many questions without answers yet, but I just wanted to raise them here.

[tssala23]

@hpdempsey @joachimweyl
Isaiah is still using Albany, and Danni is somewhat finished with Barcelona (she would like to keep the cluster up so she can reference it's current configuration but said she could also save this locally)
With that being said I think what Dylan and I do with the H100s can co exists with "test the new ACM and observability software on some cluster with updated software and H100s as well"
We will definitely need to create a new test cluster (if the new production one doesn't exist) as none of the current ones are running the latest software (we could also upgrade one but that is probably more hassle than it's worth). So that could be either tearing down barcelona and re-creating it, which is nice because we can use the same DNS records and storage allocation, or creating a completely new one.