Skip to content

Commit

Permalink
Merge pull request #1 from altoo-ag/fixcrosscompile
Browse files Browse the repository at this point in the history
Fixcrosscompile
  • Loading branch information
danischroeter authored Jul 5, 2023
2 parents abdde83 + 3e290c1 commit 946d587
Show file tree
Hide file tree
Showing 31 changed files with 136 additions and 156 deletions.
2 changes: 1 addition & 1 deletion .scalafmt.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
version = 3.6.1
runner.dialect = scala213source3
runner.dialect = scala212source3
fileOverride { ".sbt" {
runner.dialect = sbt1
} }
Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@ To use this serializer, you need to do two things:

We provide several versions of the library:

Version | Kryo Compatibility | Available Scala Versions | Tested with |
--------|--------------------|--------------------------|-----------------------------------------------------------------|
v1.0.x | Kryo-5.4 | 2.12,2.13,3.1 | JDK: OpenJdk11,OpenJdk17 Scala: 2.12.18,2.13.11,3.3.0 |
| Version | Kryo Compatibility | Available Scala Versions | Tested with |
|---------|--------------------|--------------------------|-----------------------------------------------------------------|
| v1.0.x | Kryo-5.4 | 2.12,2.13,3.1 | JDK: OpenJdk11,OpenJdk17 Scala: 2.12.18,2.13.11,3.3.0 |


Note that we use semantic versioning - see [semver.org](https://semver.org/).
Expand All @@ -65,7 +65,7 @@ To use the official release of scala-kryo-serialization in Maven projects, pleas
</snapshots>
<id>central</id>
<name>Maven Central Repository</name>
<url>http://repo1.maven.org/maven2</url>
<url>https://repo1.maven.org/maven2</url>
</repository>

<dependency>
Expand Down Expand Up @@ -107,7 +107,7 @@ You may need to repeat the process several times until you see no further log
messages about implicitly registered classes.

Another useful trick is to provide your own custom initializer for Kryo (see
below) and inside it you registerclasses of a few objects that are typically
below) and inside it, you registerclasses of a few objects that are typically
used by your application, for example:

```scala
Expand Down Expand Up @@ -275,8 +275,8 @@ like `immutable.ListMap` -- the resolver will choose the more-specific one when

`SubclassResolver` should be used with care -- even when it is turned on, you should define and
register most of your classes explicitly, as usual. But it is a helpful way to tame the complexity
of some class hierarchies, when that complexity can be treated as an implementation detail and all
of the subclasses can be serialized and deserialized identically.
of some class hierarchies, when that complexity can be treated as an implementation detail and all
the subclasses can be serialized and deserialized identically.


Using serializers with different configurations
Expand Down Expand Up @@ -315,17 +315,17 @@ Using Kryo on JDK 17
Kryo needs modules to be opened for reflection when serializing basic JDK classes.
Those options have to be passed to the JVM, for example in sbt:
```sbt
javaOptions ++= Seq("--add-opens", "java.base/java.util=ALL-UNNAMED", "--add-opens", "java.base/java.util.concurrent=ALL-UNNAMED", "--add-opens", "java.base/java.lang=ALL-UNNAMED", "--add-opens", "java.base/java.lang.invoke=ALL-UNNAMED", "--add-opens", "java.base/java.math=ALL-UNNAMED"),
javaOptions ++= Seq("--add-opens", "java.base/java.util=ALL-UNNAMED", "--add-opens", "java.base/java.util.concurrent=ALL-UNNAMED", "--add-opens", "java.base/java.lang=ALL-UNNAMED", "--add-opens", "java.base/java.lang.invoke=ALL-UNNAMED", "--add-opens", "java.base/java.math=ALL-UNNAMED")
```

To use unsafe transformations, the following access must be granted:
```sbt
javaOptions ++= Seq("--add-opens", "java.base/java.nio=ALL-UNNAMED", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED"),
javaOptions ++= Seq("--add-opens", "java.base/java.nio=ALL-UNNAMED", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED")
```

How do I build this library on my own?
--------------------------------------
If you wish to build the library on your own, you need to check out the project from Github and do
If you wish to build the library on your own, you need to check out the project from GitHub and do
```
sbt compile publishM2
```
Expand Down
3 changes: 2 additions & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ lazy val root: Project = project.in(file("."))
.settings(publish / skip := true)
.aggregate(core)

lazy val core: Project = Project("core", file("core"))
lazy val core: Project = project.in(file("core"))
.settings(moduleSettings)
.settings(description := "pekko-serialization implementation using kryo - core implementation")
.settings(name := "scala-kryo-serialization")
Expand Down Expand Up @@ -88,6 +88,7 @@ lazy val scalacBasicOptions = Seq(
"-language:existentials",
"-Xlog-reflective-calls",
"-Ywarn-unused:-nowarn",
"-Xsource:3",
"-opt:l:inline",
"-opt-inline-from:io.altoo.pekko.serialization.kryo.*")
case "3" =>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ private[kryo] object ScalaVersionSerializers {
}

def iterable(kryo: Kryo): Unit = {
kryo.addDefaultSerializer(classOf[scala.collection.Iterable[_]], classOf[ScalaCollectionSerializer])
kryo.addDefaultSerializer(classOf[scala.collection.Iterable[?]], classOf[ScalaCollectionSerializer])
}

def enums(kryo: Kryo): Unit = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,26 +26,25 @@ import com.esotericsoftware.kryo.{Kryo, Serializer}
*
* @author romix
*/
class ScalaCollectionSerializer() extends Serializer[Iterable[_]] {
class ScalaCollectionSerializer() extends Serializer[Iterable[?]] {

override def read(kryo: Kryo, input: Input, typ: Class[_ <: Iterable[_]]): Iterable[_] = {
override def read(kryo: Kryo, input: Input, typ: Class[? <: Iterable[?]]): Iterable[?] = {
val len = input.readInt(true)
val inst = kryo.newInstance(typ)
val coll = inst.iterableFactory.newBuilder[Any]

var i = 0
while (i < len) {
while i < len do {
coll += kryo.readClassAndObject(input)
i += 1
}
coll.result()
}

override def write(kryo: Kryo, output: Output, obj: Iterable[_]): Unit = {
val collection: Iterable[_] = obj
override def write(kryo: Kryo, output: Output, obj: Iterable[?]): Unit = {
val collection: Iterable[?] = obj
val len = collection.size
output.writeInt(len, true)
collection.foreach { (e: Any) => kryo.writeClassAndObject(output, e) }
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ import com.esotericsoftware.kryo.{Kryo, Serializer}
import scala.runtime.EnumValue

/** Serializes enums using the enum's name. This prevents invalidating previously serialized bytes when the enum order changes */
class ScalaEnumNameSerializer[T <: EnumValue] extends Serializer[T] {
class ScalaEnumNameSerializer[T <: EnumValue] extends Serializer[T] {

def read(kryo: Kryo, input: Input, typ: Class[_ <: T]): T = {
def read(kryo: Kryo, input: Input, typ: Class[? <: T]): T = {
val clazz = kryo.readClass(input).getType
val name = input.readString()
// using value instead of ordinal to make serialization more stable, e.g. allowing reordering without breaking compatibility
Expand All @@ -18,7 +18,7 @@ class ScalaEnumNameSerializer[T <: EnumValue] extends Serializer[T] {
def write(kryo: Kryo, output: Output, obj: T): Unit = {
val enumClass = obj.getClass.getSuperclass
val productPrefixMethod = obj.getClass.getDeclaredMethod("productPrefix")
if (!productPrefixMethod.canAccess(obj)) productPrefixMethod.setAccessible(true)
if !productPrefixMethod.canAccess(obj) then productPrefixMethod.setAccessible(true)
val name = productPrefixMethod.invoke(obj).asInstanceOf[String]
kryo.writeClass(output, enumClass)
output.writeString(name)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ class DefaultKeyProvider {
val spec = new PBEKeySpec(password.toCharArray, salt.getBytes("UTF-8"), 65536, 256)
factory.generateSecret(spec).getEncoded
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import scala.reflect.ClassTag
* you could use for example a bounded non-blocking queue.
*/
class DefaultQueueBuilder {

/**
* Override to use a different queue.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,13 +122,13 @@ private[kryo] abstract class KryoSerializer(config: Config, classLoader: ClassLo
case "lz4" => Some(new LZ4KryoCompressor)
case "deflate" => Some(new ZipKryoCompressor)
case "aes" => settings.encryptionSettings match {
case Some(es) if es.aesMode.contains("GCM") =>
Some(new KryoCryptographer(aesKeyProviderClass.get.getDeclaredConstructor().newInstance().aesKey(config.getConfig(configKey)), es.aesMode, es.aesIvLength))
case Some(es) =>
throw new Exception(s"Mode ${es.aesMode} is not supported for 'aes'")
case None =>
throw new Exception("Encryption transformation selected but encryption has not been configured")
}
case Some(es) if es.aesMode.contains("GCM") =>
Some(new KryoCryptographer(aesKeyProviderClass.get.getDeclaredConstructor().newInstance().aesKey(config.getConfig(configKey)), es.aesMode, es.aesIvLength))
case Some(es) =>
throw new Exception(s"Mode ${es.aesMode} is not supported for 'aes'")
case None =>
throw new Exception("Encryption transformation selected but encryption has not been configured")
}
case x => throw new Exception(s"Could not recognise the transformer: [$x]")
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@ import com.esotericsoftware.kryo.unsafe.{UnsafeInput, UnsafeOutput}
import org.slf4j.Logger

import java.nio.ByteBuffer
import scala.util.Success

private[kryo] class KryoSerializerBackend(val kryo: Kryo, val bufferSize: Int, val maxBufferSize: Int, val useManifest: Boolean, val useUnsafe: Boolean)(log: Logger,
classLoader: ClassLoader) {
classLoader: ClassLoader) {

// "toBinary" serializes the given object to an Array of Bytes
// Implements Serializer
Expand Down Expand Up @@ -71,7 +70,7 @@ private[kryo] class KryoSerializerBackend(val kryo: Kryo, val bufferSize: Int, v
if (useManifest) {
val clazz = manifest.flatMap(ReflectionHelper.getClassFor(_, classLoader).toOption)
clazz match {
case Some(c) => kryo.readObject(buffer, c).asInstanceOf[AnyRef]
case Some(c) => kryo.readObject(buffer, c)
case _ => throw new RuntimeException("Object of unknown class cannot be deserialized")
}
} else
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package io.altoo.serialization.kryo.scala
import scala.util.Try

object ReflectionHelper {
def getClassFor(fqcn: String, classLoader: ClassLoader): Try[Class[_ <: AnyRef]] =
Try[Class[_ <: AnyRef]] {
Class.forName(fqcn, false, classLoader).asInstanceOf[Class[_ <: AnyRef]]
def getClassFor(fqcn: String, classLoader: ClassLoader): Try[Class[? <: AnyRef]] =
Try[Class[? <: AnyRef]] {
Class.forName(fqcn, false, classLoader).asInstanceOf[Class[? <: AnyRef]]
}
}
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
package io.altoo.serialization.kryo.scala

import com.esotericsoftware.kryo.Kryo
import com.typesafe.config.Config
import io.altoo.serialization.kryo.scala.serializer.ScalaKryo

import java.nio.ByteBuffer
import scala.reflect.{ClassTag, classTag}
import scala.util.Try

/**
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package io.altoo.serialization.kryo.scala


/**
* Returns a SerializerPool, useful to reduce GC overhead.
*
Expand All @@ -14,7 +13,7 @@ private[kryo] class SerializerPool(queueBuilder: DefaultQueueBuilder, newInstanc
def fetch(): KryoSerializerBackend = {
pool.poll() match {
case null => newInstance()
case o => o
case o => o
}
}

Expand All @@ -26,4 +25,3 @@ private[kryo] class SerializerPool(queueBuilder: DefaultQueueBuilder, newInstanc
pool.add(o)
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ private[kryo] class KryoTransformer(transformations: List[Transformer]) {
private[this] val fromPipeLine = transformations.map(x => x.fromBinary(_: Array[Byte])).reverse.reduceLeftOption(_.andThen(_)).getOrElse(identity(_: Array[Byte]))

private[this] val toBufferPipeline: (Array[Byte], ByteBuffer) => Unit = transformations match {
case Nil => (_, _) => throw new UnsupportedOperationException("Should be optimized away")
case Nil => (_, _) => throw new UnsupportedOperationException("Should be optimized away")
case transformer :: Nil => transformer.toBinary
case transformations =>
val pipeline = transformations.init.map(x => x.toBinary(_: Array[Byte])).reduceLeft(_.andThen(_))
Expand All @@ -24,7 +24,7 @@ private[kryo] class KryoTransformer(transformations: List[Transformer]) {
}

private[this] val fromBufferPipeline: ByteBuffer => Array[Byte] = transformations match {
case Nil => _ => throw new UnsupportedOperationException("Should be optimized away")
case Nil => _ => throw new UnsupportedOperationException("Should be optimized away")
case transformer :: Nil => transformer.fromBinary
case transformations =>
val pipeline = transformations.init.reverse.map(x => x.fromBinary(_: Array[Byte])).reduceLeft(_.andThen(_))
Expand All @@ -51,12 +51,10 @@ private[kryo] class KryoTransformer(transformations: List[Transformer]) {
}
}


trait Transformer {
def toBinary(inputBuff: Array[Byte]): Array[Byte]

def toBinary(inputBuff: Array[Byte], outputBuff: ByteBuffer): Unit
= outputBuff.put(toBinary(inputBuff))
def toBinary(inputBuff: Array[Byte], outputBuff: ByteBuffer): Unit = outputBuff.put(toBinary(inputBuff))

def fromBinary(inputBuff: Array[Byte]): Array[Byte]

Expand All @@ -78,10 +76,10 @@ class LZ4KryoCompressor extends Transformer {
val outputSize = lz4.compress(inputBuff, 0, inputSize, outputBuff, 4, maxOutputSize)

// encode 32 bit length in the first bytes
outputBuff(0) = (inputSize & 0xff).toByte
outputBuff(1) = (inputSize >> 8 & 0xff).toByte
outputBuff(2) = (inputSize >> 16 & 0xff).toByte
outputBuff(3) = (inputSize >> 24 & 0xff).toByte
outputBuff(0) = (inputSize & 0xFF).toByte
outputBuff(1) = (inputSize >> 8 & 0xFF).toByte
outputBuff(2) = (inputSize >> 16 & 0xFF).toByte
outputBuff(3) = (inputSize >> 24 & 0xFF).toByte
outputBuff.take(outputSize + 4)
}

Expand Down Expand Up @@ -114,10 +112,10 @@ class ZipKryoCompressor extends Transformer {
val deflater = new Deflater(Deflater.BEST_SPEED)
val inputSize = inputBuff.length
val outputBuff = new mutable.ArrayBuilder.ofByte
outputBuff += (inputSize & 0xff).toByte
outputBuff += (inputSize >> 8 & 0xff).toByte
outputBuff += (inputSize >> 16 & 0xff).toByte
outputBuff += (inputSize >> 24 & 0xff).toByte
outputBuff += (inputSize & 0xFF).toByte
outputBuff += (inputSize >> 8 & 0xFF).toByte
outputBuff += (inputSize >> 16 & 0xFF).toByte
outputBuff += (inputSize >> 24 & 0xFF).toByte

deflater.setInput(inputBuff)
deflater.finish()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,21 @@ import java.lang.reflect.Field
*/
class EnumerationNameSerializer extends Serializer[Enumeration#Value] {

def read(kryo: Kryo, input: Input, typ: Class[_ <: Enumeration#Value]): Enumeration#Value = {
def read(kryo: Kryo, input: Input, typ: Class[? <: Enumeration#Value]): Enumeration#Value = {
val clazz = kryo.readClass(input).getType
val name = input.readString()
clazz.getDeclaredField("MODULE$").get(null).asInstanceOf[Enumeration].withName(name)
}

def write(kryo: Kryo, output: Output, obj: Enumeration#Value): Unit = {
val parentEnum = parent(obj.getClass.getSuperclass)
.getOrElse(throw new NoSuchElementException(s"Enumeration not found for $obj"))
.getOrElse(throw new NoSuchElementException(s"Enumeration not found for $obj"))
val enumClass = parentEnum.get(obj).getClass
kryo.writeClass(output, enumClass)
output.writeString(obj.toString)
}

private def parent(typ: Class[_]): Option[Field] =
private def parent(typ: Class[?]): Option[Field] =
if (typ == null) None
else typ.getDeclaredFields.find(_.getName == "$outer").orElse(parent(typ.getSuperclass))
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ import com.esotericsoftware.kryo.Registration
import com.esotericsoftware.kryo.util.DefaultClassResolver

class KryoClassResolver(val logImplicits: Boolean) extends DefaultClassResolver {
override def registerImplicit(typ: Class[_]): Registration = {
override def registerImplicit(typ: Class[?]): Registration = {
if (kryo.isRegistrationRequired) {
throw new IllegalArgumentException("Class is not registered: " + typ.getName
+ "\nNote: To register this class use: kryo.register(" + typ.getName + ".class);")
+ "\nNote: To register this class use: kryo.register(" + typ.getName + ".class);")
}
// registerInternal(new Registration(typ, kryo.getDefaultSerializer(typ), DefaultClassResolver.NAME))
/* TODO: This does not work if sender and receiver are
* initialized independently and using different order of classes
* Try to ensure that the same ID is assigned to the same classname
* Try to ensure that the same ID is assigned to the same classname
* by every Kryo instance:
*/
// Take a next available ID
Expand Down Expand Up @@ -79,7 +79,7 @@ class KryoClassResolver(val logImplicits: Boolean) extends DefaultClassResolver
*/
object MurmurHash {
def hash(data: Array[Byte], seed: Int): Int = {
val m: Int = 0x5bd1e995
val m: Int = 0x5BD1E995
val r: Int = 24

var h: Int = seed ^ data.length
Expand All @@ -92,11 +92,11 @@ object MurmurHash {
val i_4 = i << 2
var k: Int = data(i_4 + 3)
k = k << 8
k = k | (data(i_4 + 2) & 0xff)
k = k | (data(i_4 + 2) & 0xFF)
k = k << 8
k = k | (data(i_4 + 1) & 0xff)
k = k | (data(i_4 + 1) & 0xFF)
k = k << 8
k = k | (data(i_4 + 0) & 0xff)
k = k | (data(i_4 + 0) & 0xFF)
k *= m
k ^= k >>> r
k *= m
Expand Down
Loading

0 comments on commit 946d587

Please sign in to comment.