I haven't found the Layout system to be lacking or too difficult to deal with. My only issue with it is how verbose it feels sometimes. The operation of layout and layoutData can be a bit confusing.
You can always roll your own layout system using Control setBounds and moveAbove. I have toyed around with a custom layout system on top of SWT that uses Cassowary Constraints (in fact the imgur link in my other comment shows a sample).
As for custom widgets it really depends on how you want to do it since there are so many options. SWT can embed Swing or JavaFX components, so it allows the developer to make custom widgets in whichever tech they feel most comfortable. In this case I would stick with Swing since JavaFx can be a decently sized dependency, where as Swing is built into the jre.
The eclipse nebula project is also a pretty big repository of community made custom widgets. These can be really helpful for figuring out how to implement custom widgets in pure SWT.
https://github.com/eclipse/nebula
I have never implemented automated testing with it though, bummer it was such a hassle.
Also SWT does grant access to a bit more of the underlying OS than Swing does with its org.eclipse.swt.internal.* package. For example I've had to embed a GLFW window inside a SWT window. This is done with the following api in GLFW: https://www.glfw.org/docs/3.3/group__native.html
Using the internal apis you to map GLFWs native window handle to something SWT understands pretty easily. Then it's just a matter of writing the platform specific code to reparent the GLFW window into App which was 3 or 4 lines of code per platform. Then once the GLFW app was inside the SWT App it was a matter of creating a dummy Composite that when resized would overlay the GLFW app. I don't believe this would have been possible in Swing without addition JNI. I know this is a pretty obscure use case, but the internal package can be extremely handy.
Lastly, SWT has a built in browser component that uses the OSs browser or there are optional embedded V8 and webkit packages. The feature set of SWT just blows other Java toolkits out of the water. Unfortunately, this does lead it to be a bit more complicated than the others. I'm sorry about your experience, but give it another whirl sometime maybe you'll be pleasantly surprised.