Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Home Database Cache : Stripped Back Solution #1235

Merged
merged 84 commits into from
Jan 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
0ea5012
Migrating work from old branch
MaxAake Oct 16, 2024
c62cdaa
function with driver auth tokens as well as session auth
MaxAake Oct 17, 2024
6e9b495
minor fixes to integration and unit tests
MaxAake Oct 18, 2024
c307fb8
Update driver.test.js
MaxAake Oct 18, 2024
24cf859
expanded callback usage and fixed tests for old servers
MaxAake Oct 18, 2024
9066f24
change checks for when to update cache
MaxAake Oct 21, 2024
ca50142
Save round trip when running transactions too
MaxAake Oct 22, 2024
886b90a
update to homedbtable
MaxAake Oct 24, 2024
b4b1a48
deno
MaxAake Oct 24, 2024
f38a5df
Continued work, hard to clear cache on error :/
MaxAake Oct 25, 2024
195f05f
continued work, requires server side fix to commit success message
MaxAake Oct 29, 2024
317e282
deno
MaxAake Oct 29, 2024
d075328
logic to turn off and on caching using connection hints
MaxAake Oct 30, 2024
81a4924
deno
MaxAake Oct 30, 2024
ac11804
change back to caching database name
MaxAake Oct 31, 2024
458a493
bolt 5.8 plus reworks
MaxAake Nov 1, 2024
6b238fa
deno and minor fix
MaxAake Nov 1, 2024
936233f
remove debug log and fix minor issue
MaxAake Nov 4, 2024
a84543e
many small rewrites, test updates
MaxAake Nov 7, 2024
ced7c46
deno and removal off cachekey in hello
MaxAake Nov 7, 2024
fafde75
remove cachekey from logon and init
MaxAake Nov 7, 2024
b8633aa
protect from undefined callback
MaxAake Nov 7, 2024
2cc2e89
Update index.js
MaxAake Nov 7, 2024
050efaa
Update auth.test.js
MaxAake Nov 8, 2024
19e4f36
move hash function to allow imports
MaxAake Nov 11, 2024
dfbdeb4
deno
MaxAake Nov 11, 2024
a8a675f
small rework
MaxAake Nov 11, 2024
ae82924
add bolt 5x8 feature
MaxAake Nov 11, 2024
fde547f
cleanup and fixing for testkit tests
MaxAake Nov 14, 2024
ed2b494
deno
MaxAake Nov 14, 2024
22c795b
fixing silly self-sabotage
MaxAake Nov 14, 2024
0a39014
deno
MaxAake Nov 14, 2024
428d716
remove lingering debug log and fix unit test
MaxAake Nov 15, 2024
aab842c
Update index.test.ts
MaxAake Nov 15, 2024
ab44877
identify scheme in cache key to ensure default key is unique
MaxAake Nov 15, 2024
c3dbbc3
deno
MaxAake Nov 15, 2024
d32cd01
capped homedb cache size
MaxAake Nov 15, 2024
7bdcdd1
deno
MaxAake Nov 15, 2024
6838806
add parameters to cachekey
MaxAake Nov 18, 2024
dc90e16
deno
MaxAake Nov 18, 2024
3de0474
Update auth.test.ts
MaxAake Nov 21, 2024
28d3d8f
create Cache class and refine pruning logic
MaxAake Nov 29, 2024
521836c
refine lastdate and fix test
MaxAake Nov 29, 2024
9e92450
remove debug logging
MaxAake Dec 3, 2024
06db084
some test fixes
MaxAake Dec 3, 2024
5786c29
expansion of unit tests
MaxAake Dec 4, 2024
0ef51c0
integration tests on driver, session and impersonated auth
MaxAake Dec 4, 2024
8d75497
check for community edition in test
MaxAake Dec 5, 2024
64a3bd1
better check for impersonation and correct protocol version
MaxAake Dec 9, 2024
ab87195
fix typo in tests
MaxAake Dec 9, 2024
a5e97a8
Update driver.test.js
MaxAake Dec 9, 2024
432a34b
improved documentation
MaxAake Dec 10, 2024
cdb6778
deno
MaxAake Dec 10, 2024
968b449
clean up
MaxAake Dec 10, 2024
d40daf8
post-cleanup denobuild
MaxAake Dec 10, 2024
47429aa
correction
MaxAake Dec 11, 2024
ca2fef3
Update testkit.json
MaxAake Dec 12, 2024
38e2838
remove failures for read fails, add testkit flag
MaxAake Dec 17, 2024
734ed23
addressing comments on the PR
MaxAake Jan 7, 2025
600f028
continued addressing of reviews
MaxAake Jan 8, 2025
837ba63
unit tests and renamed auth-util file
MaxAake Jan 8, 2025
451714f
deno
MaxAake Jan 8, 2025
27c9711
import issue in unit tests
MaxAake Jan 9, 2025
e404900
remove removeFailureFromCache callback
MaxAake Jan 9, 2025
f7df396
restrict onDb callback to only bolt 5.8 run success
MaxAake Jan 9, 2025
b3c7e9e
upgrade cacheKey function to implement impersonation and default key
MaxAake Jan 9, 2025
4f33599
deno
MaxAake Jan 9, 2025
452a09e
fixing issue with correctly identifying the lack of an impersonated user
MaxAake Jan 10, 2025
718e4cd
fallback logic for missing ssr hint on new connection
MaxAake Jan 14, 2025
09f5377
skips timeout test and adds HomeDatabaseCache flag
MaxAake Jan 14, 2025
6192d6e
Update testkit.json
MaxAake Jan 16, 2025
7148dda
remove some unnecessary checks
MaxAake Jan 20, 2025
0a96302
deno
MaxAake Jan 20, 2025
254ae4d
correct bolt agent numbering
MaxAake Jan 20, 2025
0e47227
Update bolt-protocol-v5x8.test.js
MaxAake Jan 20, 2025
f7de064
minor fixes and collision free cachekeys
MaxAake Jan 22, 2025
7c0eee0
deno
MaxAake Jan 22, 2025
22b4a75
better solution
MaxAake Jan 22, 2025
ef3add5
deno
MaxAake Jan 22, 2025
5e66a89
switch to prewritten stringify
MaxAake Jan 23, 2025
4c3a739
correct expect-error-message
MaxAake Jan 23, 2025
18b3860
Moves sorting of JSON elements to stringify function.
MaxAake Jan 23, 2025
4331f70
deno
MaxAake Jan 23, 2025
e034313
formatting change
MaxAake Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/bolt-connection/src/bolt/bolt-protocol-v5x6.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
*/
import BoltProtocolV5x5 from './bolt-protocol-v5x5'

import transformersFactories from './bolt-protocol-v5x5.transformer'
import transformersFactories from './bolt-protocol-v5x6.transformer'
import Transformer from './transformer'

import { internal } from 'neo4j-driver-core'
Expand Down
2 changes: 1 addition & 1 deletion packages/bolt-connection/src/bolt/bolt-protocol-v5x7.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
*/
import BoltProtocolV5x6 from './bolt-protocol-v5x6'

import transformersFactories from './bolt-protocol-v5x5.transformer'
import transformersFactories from './bolt-protocol-v5x7.transformer'
import Transformer from './transformer'

import { internal } from 'neo4j-driver-core'
Expand Down
104 changes: 104 additions & 0 deletions packages/bolt-connection/src/bolt/bolt-protocol-v5x8.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
/**
* Copyright (c) "Neo4j"
* Neo4j Sweden AB [https://neo4j.com]
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import BoltProtocolV5x7 from './bolt-protocol-v5x7'

import transformersFactories from './bolt-protocol-v5x8.transformer'
import Transformer from './transformer'
import RequestMessage from './request-message'
import { ResultStreamObserver } from './stream-observers'

import { internal } from 'neo4j-driver-core'

const {
constants: { BOLT_PROTOCOL_V5_8, FETCH_ALL }
} = internal

export default class BoltProtocol extends BoltProtocolV5x7 {
get version () {
return BOLT_PROTOCOL_V5_8
}

get transformer () {
MaxAake marked this conversation as resolved.
Show resolved Hide resolved
if (this._transformer === undefined) {
this._transformer = new Transformer(Object.values(transformersFactories).map(create => create(this._config, this._log)))
}
return this._transformer
}

run (
query,
parameters,
{
bookmarks,
txConfig,
database,
mode,
impersonatedUser,
notificationFilter,
beforeKeys,
afterKeys,
beforeError,
afterError,
beforeComplete,
afterComplete,
flush = true,
reactive = false,
fetchSize = FETCH_ALL,
highRecordWatermark = Number.MAX_VALUE,
lowRecordWatermark = Number.MAX_VALUE,
onDb
} = {}
) {
const observer = new ResultStreamObserver({
server: this._server,
reactive,
fetchSize,
moreFunction: this._requestMore.bind(this),
discardFunction: this._requestDiscard.bind(this),
beforeKeys,
afterKeys,
beforeError,
afterError,
beforeComplete,
afterComplete,
highRecordWatermark,
lowRecordWatermark,
enrichMetadata: this._enrichMetadata,
onDb
})

const flushRun = reactive
this.write(
RequestMessage.runWithMetadata5x5(query, parameters, {
bookmarks,
txConfig,
database,
mode,
impersonatedUser,
notificationFilter
}),
observer,
flushRun && flush
)

if (!reactive) {
this.write(RequestMessage.pull({ n: fetchSize }), observer, flush)
}

return observer
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/**
* Copyright (c) "Neo4j"
* Neo4j Sweden AB [https://neo4j.com]
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import v5x7 from './bolt-protocol-v5x7.transformer'

export default {
...v5x7
}
9 changes: 9 additions & 0 deletions packages/bolt-connection/src/bolt/create.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import BoltProtocolV5x4 from './bolt-protocol-v5x4'
import BoltProtocolV5x5 from './bolt-protocol-v5x5'
import BoltProtocolV5x6 from './bolt-protocol-v5x6'
import BoltProtocolV5x7 from './bolt-protocol-v5x7'
import BoltProtocolV5x8 from './bolt-protocol-v5x8'
// eslint-disable-next-line no-unused-vars
import { Chunker, Dechunker } from '../channel'
import ResponseHandler from './response-handler'
Expand Down Expand Up @@ -257,6 +258,14 @@ function createProtocol (
log,
onProtocolError,
serversideRouting)
case 5.8:
return new BoltProtocolV5x8(server,
chunker,
packingConfig,
createResponseHandler,
log,
onProtocolError,
serversideRouting)
default:
throw newError('Unknown Bolt protocol version: ' + version)
}
Expand Down
2 changes: 1 addition & 1 deletion packages/bolt-connection/src/bolt/handshake.js
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ function parseNegotiatedResponse (buffer, log) {
*/
function newHandshakeBuffer () {
return createHandshakeMessage([
[version(5, 7), version(5, 0)],
[version(5, 8), version(5, 0)],
[version(4, 4), version(4, 2)],
version(4, 1),
version(3, 0)
Expand Down
8 changes: 7 additions & 1 deletion packages/bolt-connection/src/bolt/stream-observers.js
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,8 @@ class ResultStreamObserver extends StreamObserver {
server,
highRecordWatermark = Number.MAX_VALUE,
lowRecordWatermark = Number.MAX_VALUE,
enrichMetadata
enrichMetadata,
onDb
} = {}) {
super()

Expand Down Expand Up @@ -113,6 +114,7 @@ class ResultStreamObserver extends StreamObserver {
this._paused = false
this._pulled = !reactive
this._haveRecordStreamed = false
this._onDb = onDb
}

/**
Expand Down Expand Up @@ -319,6 +321,10 @@ class ResultStreamObserver extends StreamObserver {
}
}

if (meta.db !== null && this._onDb !== undefined) {
this._onDb(meta.db)
}

if (meta.fields != null) {
// remove fields key from metadata object
delete meta.fields
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@ export default class RoutingConnectionProvider extends PooledConnectionProvider
this._createConnectionErrorHandler(),
this._log,
await this._clientCertificateHolder.getClientCertificate(),
this._routingContext
this._routingContext,
this._channelSsrCallback.bind(this)
)
})

Expand All @@ -99,6 +100,8 @@ export default class RoutingConnectionProvider extends PooledConnectionProvider
)

this._refreshRoutingTable = functional.reuseOngoingRequest(this._refreshRoutingTable, this)
this._withSSR = 0
this._withoutSSR = 0
}

_createConnectionErrorHandler () {
Expand Down Expand Up @@ -139,19 +142,30 @@ export default class RoutingConnectionProvider extends PooledConnectionProvider
* See {@link ConnectionProvider} for more information about this method and
* its arguments.
*/
async acquireConnection ({ accessMode, database, bookmarks, impersonatedUser, onDatabaseNameResolved, auth } = {}) {
let name
let address
async acquireConnection ({ accessMode, database, bookmarks, impersonatedUser, onDatabaseNameResolved, auth, homeDb } = {}) {
const context = { database: database || DEFAULT_DB_NAME }

const databaseSpecificErrorHandler = new ConnectionErrorHandler(
SESSION_EXPIRED,
(error, address) => this._handleUnavailability(error, address, context.database),
(error, address) => this._handleWriteFailure(error, address, context.database),
(error, address, conn) =>
this._handleSecurityError(error, address, conn, context.database)
(error, address) => this._handleWriteFailure(error, address, homeDb ?? context.database),
(error, address, conn) => this._handleSecurityError(error, address, conn, context.database)
)

let conn
if (this.SSREnabled() && homeDb !== undefined && database === '') {
const currentRoutingTable = this._routingTableRegistry.get(
homeDb,
() => new RoutingTable({ database: homeDb })
)
if (currentRoutingTable && !currentRoutingTable.isStaleFor(accessMode)) {
conn = await this.getConnectionFromRoutingTable(currentRoutingTable, auth, accessMode, databaseSpecificErrorHandler)
if (this.SSREnabled()) {
return conn
}
conn.release()
}
}
const routingTable = await this._freshRoutingTable({
accessMode,
database: context.database,
Expand All @@ -165,7 +179,12 @@ export default class RoutingConnectionProvider extends PooledConnectionProvider
}
}
})
return this.getConnectionFromRoutingTable(routingTable, auth, accessMode, databaseSpecificErrorHandler)
}

async getConnectionFromRoutingTable (routingTable, auth, accessMode, databaseSpecificErrorHandler) {
let name
let address
// select a target server based on specified access mode
if (accessMode === READ) {
address = this._loadBalancingStrategy.selectReader(routingTable.readers)
Expand Down Expand Up @@ -663,6 +682,28 @@ export default class RoutingConnectionProvider extends PooledConnectionProvider
routingTable.forgetRouter(address)
}
}

_channelSsrCallback (isEnabled, action) {
if (action === 'OPEN') {
if (isEnabled === true) {
this._withSSR = this._withSSR + 1
} else {
this._withoutSSR = this._withoutSSR + 1
}
} else if (action === 'CLOSE') {
if (isEnabled === true) {
this._withSSR = this._withSSR - 1
} else {
this._withoutSSR = this._withoutSSR - 1
}
} else {
throw newError("Channel SSR Callback invoked with action other than 'OPEN' or 'CLOSE'")
}
}

SSREnabled () {
return this._withSSR > 0 && this._withoutSSR === 0
}
}

/**
Expand Down
21 changes: 16 additions & 5 deletions packages/bolt-connection/src/connection/connection-channel.js
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ let idGenerator = 0
* @param {ConnectionErrorHandler} errorHandler - the error handler for connection errors.
* @param {Logger} log - configured logger.
* @param {clientCertificate} clientCertificate - configured client certificate
* @param ssrCallback - callback function used to update the counts of ssr enabled and disabled connections
* @param createChannel - function taking a channelConfig object and creating a channel with it
* @return {Connection} - new connection.
*/
export function createChannelConnection (
Expand All @@ -43,6 +45,7 @@ export function createChannelConnection (
log,
clientCertificate,
serversideRouting = null,
ssrCallback,
createChannel = channelConfig => new Channel(channelConfig)
) {
const channelConfig = new ChannelConfig(
Expand Down Expand Up @@ -89,7 +92,8 @@ export function createChannelConnection (
chunker,
config.notificationFilter,
createProtocol,
config.telemetryDisabled
config.telemetryDisabled,
ssrCallback
)

// forward all pending bytes to the dechunker
Expand All @@ -110,9 +114,11 @@ export default class ChannelConnection extends Connection {
* @param {ConnectionErrorHandler} errorHandler the error handler.
* @param {ServerAddress} address - the server address to connect to.
* @param {Logger} log - the configured logger.
* @param {boolean} disableLosslessIntegers if this connection should convert all received integers to native JS numbers.
* @param {Chunker} chunker the chunker
* @param protocolSupplier Bolt protocol supplier
* @param {boolean} disableLosslessIntegers - if this connection should convert all received integers to native JS numbers.
* @param {Chunker} chunker - the chunker
* @param protocolSupplier - Bolt protocol supplier
* @param {boolean} telemetryDisabled - wether telemetry has been disabled in driver config.
* @param ssrCallback - callback function used to update the counts of ssr enabled and disabled connections.
*/
constructor (
channel,
Expand All @@ -124,7 +130,8 @@ export default class ChannelConnection extends Connection {
chunker, // to be removed,
notificationFilter,
protocolSupplier,
telemetryDisabled
telemetryDisabled,
ssrCallback = (_) => {}
) {
super(errorHandler)
this._authToken = null
Expand All @@ -143,6 +150,7 @@ export default class ChannelConnection extends Connection {
this._notificationFilter = notificationFilter
this._telemetryDisabledDriverConfig = telemetryDisabled === true
this._telemetryDisabledConnection = true
this._ssrCallback = ssrCallback

// connection from the database, returned in response for HELLO message and might not be available
this._dbConnectionId = null
Expand Down Expand Up @@ -331,7 +339,9 @@ export default class ChannelConnection extends Connection {
if (telemetryEnabledHint === true) {
this._telemetryDisabledConnection = false
}
this.SSREnabledHint = metadata.hints['ssr.enabled']
}
this._ssrCallback(this.SSREnabledHint ?? false, 'OPEN')
}
resolve(self)
}
Expand Down Expand Up @@ -538,6 +548,7 @@ export default class ChannelConnection extends Connection {
* @returns {Promise<void>} - A promise that will be resolved when the underlying channel is closed.
*/
async close () {
this._ssrCallback(this.SSREnabledHint ?? false, 'CLOSE')
if (this._log.isDebugEnabled()) {
this._log.debug('closing')
}
Expand Down
Loading