Mercurial > vec

annotate src/impl/x86/avx2.c @ 28:c6c99ab1088a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
*: add min/max functions and a big big refactor (again) agh, this time I added a few more implementations (and generally made the code just a little faster...)
author Paper <paper@tflc.us>
date Thu, 24 Apr 2025 00:54:02 -0400
parents e49e70f7012f
children bf6ad516f1e6
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
1 /**
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
2 * vec - a tiny SIMD vector library in C99
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
3 *
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
4 * Copyright (c) 2024 Paper
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
5 *
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
6 * Permission is hereby granted, free of charge, to any person obtaining a copy
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
7 * of this software and associated documentation files (the "Software"), to deal
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
8 * in the Software without restriction, including without limitation the rights
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
9 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
10 * copies of the Software, and to permit persons to whom the Software is
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
11 * furnished to do so, subject to the following conditions:
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
12 *
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
13 * The above copyright notice and this permission notice shall be included in all
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
14 * copies or substantial portions of the Software.
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
15 *
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
16 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
17 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
18 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
19 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
20 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
21 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
22 * SOFTWARE.
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
23 **/
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
24
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
25 #include "vec/impl/x86/avx2.h"
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
26
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
27 #include <immintrin.h>
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
28
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
29 /* ------------------------------------------------------------------------ */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
30 /* min/max */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
31
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
32 #define VEC_AVX2_MINMAX_TEMPLATE(SIGN, BITS, SIZE, INTLSIGN, OP) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
33 VEC_FUNC_IMPL v##SIGN##int##BITS##x##SIZE v##SIGN##int##BITS##x##SIZE##_avx2_##OP(v##SIGN##int##BITS##x##SIZE vec1, v##SIGN##int##BITS##x##SIZE vec2) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
34 { \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
35 union v##SIGN##int##BITS##x##SIZE##_impl_data *vec1d = (union v##SIGN##int##BITS##x##SIZE##_impl_data *)&vec1; \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
36 union v##SIGN##int##BITS##x##SIZE##_impl_data *vec2d = (union v##SIGN##int##BITS##x##SIZE##_impl_data *)&vec2; \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
37 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
38 vec1d->avx2 = _mm256_##OP##_ep##INTLSIGN##BITS(vec1d->avx2, vec2d->avx2); \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
39 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
40 return vec1d->vec; \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
41 }
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
42
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
43 #define VEC_AVX2_MINMAX_8x32(OP) VEC_AVX2_MINMAX_TEMPLATE( , 8, 32, i, OP)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
44 #define VEC_AVX2_MINMAX_u8x32(OP) VEC_AVX2_MINMAX_TEMPLATE(u, 8, 32, u, OP)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
45 #define VEC_AVX2_MINMAX_16x16(OP) VEC_AVX2_MINMAX_TEMPLATE( , 16, 16, i, OP)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
46 #define VEC_AVX2_MINMAX_u16x16(OP) VEC_AVX2_MINMAX_TEMPLATE(u, 16, 16, u, OP)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
47 #define VEC_AVX2_MINMAX_32x8(OP) VEC_AVX2_MINMAX_TEMPLATE( , 32, 8, i, OP)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
48 #define VEC_AVX2_MINMAX_u32x8(OP) VEC_AVX2_MINMAX_TEMPLATE(u, 32, 8, u, OP)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
49 #define VEC_AVX2_MINMAX_64x4(OP) /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
50 #define VEC_AVX2_MINMAX_u64x4(OP) /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
51
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
52 #define VEC_AVX2_STRUCT_MINMAX_8x32(OP, SIGN) v##SIGN##int8x32_avx2_##OP
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
53 #define VEC_AVX2_STRUCT_MINMAX_16x16(OP, SIGN) v##SIGN##int16x16_avx2_##OP
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
54 #define VEC_AVX2_STRUCT_MINMAX_32x8(OP, SIGN) v##SIGN##int32x8_avx2_##OP
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
55 #define VEC_AVX2_STRUCT_MINMAX_64x4(OP, SIGN) NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
56
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
57 /* reused this for avg */
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
58
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
59 #define VEC_AVX2_AVG_8x32 /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
60 #define VEC_AVX2_AVG_16x16 /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
61 #define VEC_AVX2_AVG_32x8 /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
62 #define VEC_AVX2_AVG_64x4 /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
63
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
64 #define VEC_AVX2_AVG_u8x32 VEC_AVX2_MINMAX_TEMPLATE(u, 8, 32, u, avg)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
65 #define VEC_AVX2_AVG_u16x16 VEC_AVX2_MINMAX_TEMPLATE(u, 16, 16, u, avg)
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
66 #define VEC_AVX2_AVG_u32x8 /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
67 #define VEC_AVX2_AVG_u64x4 /* nothing */
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
68
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
69 #define VEC_AVX2_STRUCT_AVG_8x32 NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
70 #define VEC_AVX2_STRUCT_AVG_16x16 NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
71 #define VEC_AVX2_STRUCT_AVG_32x8 NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
72 #define VEC_AVX2_STRUCT_AVG_64x4 NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
73
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
74 #define VEC_AVX2_STRUCT_AVG_u8x32 vuint8x32_avx2_avg
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
75 #define VEC_AVX2_STRUCT_AVG_u16x16 vuint16x16_avx2_avg
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
76 #define VEC_AVX2_STRUCT_AVG_u32x8 NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
77 #define VEC_AVX2_STRUCT_AVG_u64x4 NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
78
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
79 /* ------------------------------------------------------------------------ */
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
80
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
81 // multiplication
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
82
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
83 #define VEC_AVX2_MUL_8x32(sign) /* nothing */
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
84
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
85 #define VEC_AVX2_MUL_16x16(sign) \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
86 VEC_FUNC_IMPL v##sign##int16x16 v##sign##int16x16_avx2_mul(v##sign##int16x16 vec1, v##sign##int16x16 vec2) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
87 { \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
88 union v##sign##int16x16_impl_data *vec1d = (union v##sign##int16x16_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
89 union v##sign##int16x16_impl_data *vec2d = (union v##sign##int16x16_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
90 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
91 vec1d->avx2 = _mm256_mullo_epi16(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
92 return vec1d->vec; \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
93 }
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
94
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
95 #define VEC_AVX2_MUL_32x8(sign) \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
96 VEC_FUNC_IMPL v##sign##int32x8 v##sign##int32x8_avx2_mul(v##sign##int32x8 vec1, v##sign##int32x8 vec2) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
97 { \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
98 union v##sign##int32x8_impl_data *vec1d = (union v##sign##int32x8_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
99 union v##sign##int32x8_impl_data *vec2d = (union v##sign##int32x8_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
100 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
101 vec1d->avx2 = _mm256_mullo_epi32(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
102 return vec1d->vec; \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
103 }
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
104
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
105 #define VEC_AVX2_MUL_64x4(sign) \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
106 VEC_FUNC_IMPL v##sign##int64x4 v##sign##int64x4_avx2_mul(v##sign##int64x4 vec1, v##sign##int64x4 vec2) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
107 { \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
108 union v##sign##int64x4_impl_data *vec1d = (union v##sign##int64x4_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
109 union v##sign##int64x4_impl_data *vec2d = (union v##sign##int64x4_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
110 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
111 __m256i ac = _mm256_mul_epu32(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
112 __m256i b = _mm256_srli_epi64(vec1d->avx2, 32); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
113 __m256i bc = _mm256_mul_epu32(b, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
114 __m256i d = _mm256_srli_epi64(vec2d->avx2, 32); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
115 __m256i ad = _mm256_mul_epu32(vec1d->avx2, d); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
116 __m256i hi = _mm256_add_epi64(bc, ad); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
117 hi = _mm256_slli_epi64(hi, 32); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
118 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
119 vec1d->avx2 = _mm256_add_epi64(hi, ac); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
120 return vec1d->vec; \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
121 }
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
122
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
123 #define VEC_AVX2_STRUCT_MUL_8x32(SIGN) NULL
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
124 #define VEC_AVX2_STRUCT_MUL_16x16(SIGN) v##SIGN##int16x16_avx2_mul
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
125 #define VEC_AVX2_STRUCT_MUL_32x8(SIGN) v##SIGN##int32x8_avx2_mul
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
126 #define VEC_AVX2_STRUCT_MUL_64x4(SIGN) v##SIGN##int64x4_avx2_mul
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
127
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
128 // operations
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
129
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
130 #define VEC_AVX2_DEFINE_OPERATIONS_SIGN(sign, bits, size) \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
131 union v##sign##int##bits##x##size##_impl_data { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
132 v##sign##int##bits##x##size vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
133 __m256i avx2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
134 }; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
135 \
24
e49e70f7012f impl/x86: add static assertions for alignment and size
Paper <paper@tflc.us>
parents: 23
diff changeset
136 VEC_STATIC_ASSERT(VEC_ALIGNOF(__m256i) <= VEC_ALIGNOF(v##sign##int##bits##x##size), "vec: v" #sign "int" #bits "x" #size " alignment needs to be expanded to fit intrinsic type size"); \
e49e70f7012f impl/x86: add static assertions for alignment and size
Paper <paper@tflc.us>
parents: 23
diff changeset
137 VEC_STATIC_ASSERT(sizeof(__m256i) <= sizeof(v##sign##int##bits##x##size), "vec: v" #sign "int" #bits "x" #size " needs to be expanded to fit intrinsic type size"); \
e49e70f7012f impl/x86: add static assertions for alignment and size
Paper <paper@tflc.us>
parents: 23
diff changeset
138 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
139 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_load_aligned(const vec_##sign##int##bits in[size]) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
140 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
141 union v##sign##int##bits##x##size##_impl_data vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
142 vec.avx2 = _mm256_load_si256((const __m256i *)in); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
143 return vec.vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
144 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
145 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
146 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_load(const vec_##sign##int##bits in[size]) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
147 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
148 union v##sign##int##bits##x##size##_impl_data vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
149 vec.avx2 = _mm256_loadu_si256((const __m256i *)in); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
150 return vec.vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
151 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
152 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
153 VEC_FUNC_IMPL void v##sign##int##bits##x##size##_avx2_store_aligned(v##sign##int##bits##x##size vec, vec_##sign##int##bits out[size]) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
154 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
155 _mm256_store_si256((__m256i *)out, ((union v##sign##int##bits##x##size##_impl_data*)&vec)->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
156 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
157 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
158 VEC_FUNC_IMPL void v##sign##int##bits##x##size##_avx2_store(v##sign##int##bits##x##size vec, vec_##sign##int##bits out[size]) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
159 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
160 _mm256_storeu_si256((__m256i *)out, ((union v##sign##int##bits##x##size##_impl_data*)&vec)->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
161 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
162 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
163 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_add(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
164 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
165 union v##sign##int##bits##x##size##_impl_data *vec1d = (union v##sign##int##bits##x##size##_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
166 union v##sign##int##bits##x##size##_impl_data *vec2d = (union v##sign##int##bits##x##size##_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
167 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
168 vec1d->avx2 = _mm256_add_epi##bits(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
169 return vec1d->vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
170 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
171 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
172 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_sub(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
173 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
174 union v##sign##int##bits##x##size##_impl_data *vec1d = (union v##sign##int##bits##x##size##_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
175 union v##sign##int##bits##x##size##_impl_data *vec2d = (union v##sign##int##bits##x##size##_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
176 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
177 vec1d->avx2 = _mm256_sub_epi##bits(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
178 return vec1d->vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
179 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
180 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
181 VEC_AVX2_MUL_##bits##x##size(sign) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
182 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
183 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_and(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
184 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
185 union v##sign##int##bits##x##size##_impl_data *vec1d = (union v##sign##int##bits##x##size##_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
186 union v##sign##int##bits##x##size##_impl_data *vec2d = (union v##sign##int##bits##x##size##_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
187 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
188 vec1d->avx2 = _mm256_and_si256(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
189 return vec1d->vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
190 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
191 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
192 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_or(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
193 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
194 union v##sign##int##bits##x##size##_impl_data *vec1d = (union v##sign##int##bits##x##size##_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
195 union v##sign##int##bits##x##size##_impl_data *vec2d = (union v##sign##int##bits##x##size##_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
196 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
197 vec1d->avx2 = _mm256_or_si256(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
198 return vec1d->vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
199 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
200 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
201 VEC_FUNC_IMPL v##sign##int##bits##x##size v##sign##int##bits##x##size##_avx2_xor(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
202 { \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
203 union v##sign##int##bits##x##size##_impl_data *vec1d = (union v##sign##int##bits##x##size##_impl_data *)&vec1; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
204 union v##sign##int##bits##x##size##_impl_data *vec2d = (union v##sign##int##bits##x##size##_impl_data *)&vec2; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
205 \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
206 vec1d->avx2 = _mm256_xor_si256(vec1d->avx2, vec2d->avx2); \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
207 return vec1d->vec; \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
208 } \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
209 \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
210 VEC_AVX2_AVG_##sign##bits##x##size \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
211 \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
212 VEC_AVX2_MINMAX_##sign##bits##x##size(min) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
213 VEC_AVX2_MINMAX_##sign##bits##x##size(max) \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
214 \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
215 const v##sign##int##bits##x##size##_impl v##sign##int##bits##x##size##_impl_avx2 = { \
28
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
216 .load_aligned = v##sign##int##bits##x##size##_avx2_load_aligned, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
217 .load = v##sign##int##bits##x##size##_avx2_load, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
218 .store_aligned = v##sign##int##bits##x##size##_avx2_store_aligned, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
219 .store = v##sign##int##bits##x##size##_avx2_store, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
220 .add = v##sign##int##bits##x##size##_avx2_add, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
221 .sub = v##sign##int##bits##x##size##_avx2_sub, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
222 .mul = VEC_AVX2_STRUCT_MUL_##bits##x##size(sign), \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
223 .band = v##sign##int##bits##x##size##_avx2_and, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
224 .bor = v##sign##int##bits##x##size##_avx2_or, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
225 .bxor = v##sign##int##bits##x##size##_avx2_xor, \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
226 .min = VEC_AVX2_STRUCT_MINMAX_##bits##x##size(min, sign), \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
227 .max = VEC_AVX2_STRUCT_MINMAX_##bits##x##size(max, sign), \
c6c99ab1088a *: add min/max functions and a big big refactor (again)
Paper <paper@tflc.us>
parents: 24
diff changeset
228 .avg = VEC_AVX2_STRUCT_AVG_##sign##bits##x##size, \
23
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
229 };
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
230
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
231 #define VEC_AVX2_DEFINE_OPERATIONS(bits, size) \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
232 VEC_AVX2_DEFINE_OPERATIONS_SIGN( , bits, size) \
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
233 VEC_AVX2_DEFINE_OPERATIONS_SIGN(u, bits, size)
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
234
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
235 VEC_AVX2_DEFINE_OPERATIONS(8, 32)
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
236 VEC_AVX2_DEFINE_OPERATIONS(16, 16)
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
237 VEC_AVX2_DEFINE_OPERATIONS(32, 8)
e26874655738 *: huge refactor, new major release (hahaha)
Paper <paper@tflc.us>
parents:
diff changeset
238 VEC_AVX2_DEFINE_OPERATIONS(64, 4)